AFIT/GCM/LAS/94S-5 


DEFECTIVE  PRICING:  AN  ANALYSIS  OF 
FACTORS  AFFECTING  SUSTENTION 
RATES  AND  DISPOSITION  TIMES 

THESIS 


Tracey  D.  Kop  Dawn  C.  Sutton 

Captain,  USAF  GS-12,USAF 

AFIT/GCM/LAS/94S-5 


Accesion  For 


NTIS  CRA&I 
DTIC  TAB  □ 

Ursannounced  q 

JiJitification 


Dist 


Availaui.-.ty  ^odes 


I  Ava-' 


It!.*;  /  or 


Special 


me  QUALETf  INSPECTED  9 


Approved  for  public  release;  distribution  unlimited 


94-31 109 

liinnH 


84  9  28  15  8 


The  views  expressed  in  this  thesis  are  those  of  the  authors 
and  do  not  reflect  the  official  policy  or  position  of  the 
Department  of  Defense  or  the  U.S.  Government. 


AFrr/GCM/LASy94S-5 


DEFECTIVE  PRICING:  AN  ANALYSIS  OF  FACTORS  AFFECTING 
SUSTENTION  RATES  AND  DISPOSITION  TIMES 


THESIS 


Presented  to  the  Faculty  of  the  Graduate  School  of 
Logistics  and  Acquisition  Management  of 
the  Air  Force  Institute  of  Technology 
Air  University 
In  Partial  Fulfilln^nt  of  the 
Requirements  for  the  Degree  of 
Master  of  Science  in  Contracting  Management 


Tracey  D.  Kop,  B.A.  Dawn  C.  Sutton,  B.A. 

Captain,  USAF  GS- 12,  USAF 


September  1994 


Approved  for  public  release;  distribution  unlimited 


We  would  like  to  thank  our  advisors,  Lt  Col  Carl  Templin  and  Major  Curtis  Britt, 
for  helping  us  define  our  research  objectives  and  keeping  us  focused  on  those  objectives. 


We  appreciate  the  encouragement  and  suggestions  throughout  the  entire  process, 
especially  during  the  last  hectic  month.  We  also  thank  Mr.  Eugene  Solimine  for 
sponsoring  this  research  and  Dr.  David  Barr  for  guiding  us  through  the  perils  of  statistical 
analysis. 

This  research  would  not  have  been  possible  without  the  assistance  of  the  defective 
pricing  focal  points,  Maj  Henry  Gaudreau,  Capt  Gary  Shafovaloff,  Ms.  Pat  Herrick,  and 
Mr.  Paul  Guntarz,  who  provided  access  to  their  contract  files.  We  are  also  indebted  to 
Captain  Gecffge  Caltabellotta  for  providing  the  many  data  printouts  necessary  to  complete 
this  effort 

We  are  grateful  to  our  families  for  providing  some  quiet  time  so  that  we  could 
concentrate  and  complete  this  endeavor.  We  are  also  grateful  that  we  were  allowed  to 
have  a  partner  with  whom  to  share  this  roller  coaster  ride. 

T.  Kop  &  D.  Sutton 


n 


Table  of  Contents 


Page 

Acknowledgments . ii 

List  of  Figures . vi 

List  of  Tables . vii 

Abstract . ix 

L  Introduction . 1 

Background . 1 

Disposition  Times . 2 

Sustention  Rates . 3 

Specific  Problem . 3 

Research  Objectives . 4 

Scope . 4 

Overview  of  Subsequent  Chtq)ters . 4 

n.  Literature  Review . 6 

Overview . 6 

Truth  in  Negotiations  Act  (TINA) . 6 

Amendments . 8 

Phase  1:  Burden  of  Proof . 8 

Proof  Element  1:  Defective  Data  are  Cost  or  Pricing  Data . 9 

Proof  Element  2:  Cost  or  Pricing  Data  Existed  and  Were  Available . 10 

Proof  Element  3:  Cost  or  Pricing  Data  Were  Not  Submitted . 1 1 

Proof  Element  4;  Government  Relied  of  Defective  Data . 11 

Proof  Element  5:  Reliance  on  Defective  Data  Caused  an  Increase  in 

Contract  Price . 12 

Phase  2:  Quantifying  the  Defect  Amount . 12 

Disposition  Process . 13 

Detection . 13 

Resolution . 14 

Critical  Policy  Issues . 15 

Distinction  Between  Defective  Pricing  and  Fraud . 15 

Subcontracts . 16 

Sweep  Data . 16 

Interest  Payments  and  Penalty  Provisions . 17 

iii 


Page 


Factors  Suggested  by  Oversight  Organizations . 18 

Low  Sustention  Rates . 18 

Lengthy  Disposition  Times . 19 

Past  Research . 20 

Identification  of  Factors  from  Literature  Review . 21 

Summary . 23 

in.  Methodology . 24 

Overview . 24 

Identificadon  of  Factors  from  Interviews . 24 

Method  Selection . 24 

Survey  Development . 25 

Participant  Selection . 25 

Interview  Technique . 26 

Data  Collection . 27 

PK  Audit  Reporting  System  (PARS) . 27 

Ccmtract  Files . 28 

Data  Selection . 28 

Analysis  of  Variance . 29 

ANOVA  Models . 30 

Hypothesis  Testing . 30 

Coefficient  of  Determination . 31 

Assumptions . 31 

Model  Building . 32 

Multiple  Linear  Regression  (MLR) . 32 

Step  1:  Best  Subset  Selection . 33 

Step  2:  Verification  of  Model  Assumptions . 34 

Residual  Analysis . 34 

Heteroscedasticity . 34 

Normal  Distribution  of  Residuals . 34 

Multicollinearity . 35 

Step  3:  Model  Determination  and  Evaluation . 35 

F-Test . 35 

Coefficient  of  Determination . 35 

Step  4:  Model  Validation . 36 

Summary . 36 

rv.  Results . 37 

Overview . 37 

Interview  Results . 37 


IV 


Page 

Analysis  of  Variance . 41 

Summary  of  Analysis  of  Variance  Results . 57 

Model  Building . 60 

Best  Subset  Selection . 61 

Verification  of  Model  Assumptions . 62 

Heteroscedasticity . 62 

Normal  Distribution  of  Residuals . 66 

Multicollinearity . 68 

Model  Determination  and  Evaluation . 68 

Model  Validation . 71 

V.  Summary  and  Conclusions . 72 

Summary . 72 

Research  Objectives . 72 

Methodology  and  Summary  Results . 73 

Research  Objectives  1  and  2 . 73 

Research  Objective  3 . 73 

Research  Objectives  4  and  5 . 74 

Conclusions  and  Findings  for  Sustention  Rates . 77 

Conclusions  and  Findings  for  Disposition  Times . 81 

Additional  Conclusions  and  Findings . 84 

Recommendations  for  Improving  Sustention  Rates . 86 

Recommendations  for  Improving  Disposition  Times . 88 

Suggestions  for  Further  Research . 89 

Appendix  A:  Defective  Pricing  Survey . 90 

Appendix  B:  List  of  Interview  Participants . 94 

Appendix  C:  PARS  Data  Sheets . 95 

Appendix  D:  Additional  Factors  Suggested  During  Interviews . 107 

Appendix  E:  Variable  Descriptions . 1 10 

Appendix  F:  Summary  of  Subset  Selection  Results  Using  Miller's  Method . 1 13 

Bibliography . 114 

Vitae . 117 


V 


List  of  Figures 


Figure  Page 

1.  Residual  Plot  for  Sustention  Rates . 63 

2.  Residual  Plot  fOT  Disposition  Times . 63 

3.  Residual  Plot  fw  Transformed  Disposition  Times . 65 

4.  Wilk-Shapiro/Rankit  Plot  for  Transformed  Disposition  Times . 66 

5.  Wilk-ShapmVRankit  Plot  for  Sustention  Rates . 67 

6.  Histogram  of  Studentized  Residuals  for  Sustention  Rates . 68 


VI 


Table 


Page 


1.  Summary  of  Suggested  Factors  for  Sustention  Rates  and  Disposition  Times . 21 

2.  Interview  Results . 39 

3.  Alleged  Defect  Amount  -  ANOVA  Results . 42 

4.  Alleged  Defect  Amount  -  Mean  Comparisons . 43 

5.  Cost  Element  -  ANOVA  Results . 44 

6.  Contractor  Level  -  ANOVA  Results . 44 

7.  Contractor  Level  -  Mean  Comparisons . 45 

8.  CO  Disagreement  with  RPA  -  ANOVA  Results . 46 

9.  CO  Disagreement  with  RPA  -  Mean  Comparisons . . . 46 

10.  Number  of  Issues  -  ANOVA  Results . 47 

1 1 .  Number  of  Issues  -  Mean  Comparisons . 47 

12.  Legal  Ctraaplexity  -  ANOVA  Results . 48 

13.  Legal  Complexity  -  Mean  Comparisons . 48 

14.  Offsets  -  ANOVA  Results . 49 

15.  Offsets  -  Mean  Con:q)arisons . 49 

1 6.  Method  of  Disposition  -  ANOV  A  Results . 50 

17.  Method  of  Disposition  -  Mean  Comparisons . 50 

18.  Fraud  Investigation  -  ANOVA  Results . 51 

19.  Disposition  Time  -  ANOVA  Results . 52 

20.  Disposition  Time  -  Mean  Comparisons . 52 

21.  Identity  of  Prime  Contractor  -  ANOVA  Results . 53 

22.  Identity  of  Prime  Contractor  -  Mean  Comparisons . 54 

23.  Product  Center  -  ANOVA  Results . 54 


vii 


Table  Page 

24.  Product  Center  -  Mean  Comparisons . 55 

25.  Interest  Assessed  -  A.40VA  Results . 55 

26.  Interest  Assessed  -  Mean  CcHnparisons . 56 

27.  RPA  RevisiOTs  -  ANOVA  Results . 56 

28.  RPA  Revisions  -  Mean  Comparisons . 57 

29.  Sustention  Rate  Mean  Comparisons . 58 

30.  Disposition  Time  Mean  Comparisons . 59 

31.  Factors  Considered  in  MLR  Models . 60 

32.  Linear  Regression  Results  for  Sustention  Rates . 61 

33.  Linear  Regression  Results  for  Disposition  Times . 62 

34.  Linear  Regression  Results  fa*  Disposition  limes  with  Logarithmic 

Transformation . 65 

35.  Predictor  Variables  Used  in  MLR  Models . 69 

36.  Comparison  of  Fitted  R^  and  Predictive  R^ . 7 1 

37.  Results  for  Research  Objectives  1, 2,  arvl  3 . 74 

38.  Predictor  Variables  for  Regression  Models . 76 


viii 


AFIT/GCM/LASy94S-5 


Abstract 

Defective  pricing  occurs  when  contractors  fail  to  disclose  current,  accurate,  and 
complete  cost  or  pricing  data  in  their  proposals.  Failure  to  submit  valid  data  entitles  the 
govemnwnt  to  a  refund  in  the  amount  of  overpayment  With  the  current  backlog  of 
overage  defective  pricing  cases  and  the  continuing  decline  in  sustention  rates,  a  better 
understanding  of  the  factors  affecting  timely  and  successful  recoupment  of  defective 
pricing  funds  is  needed.  This  research  identifies  factors  which  significandy  affect 
sustention  rates  and  disposition  times  and  presents  models  to  predict  both  rates  and  times. 
Factors  were  identified  through  a  literature  review  and  interviews  with  defective  pricing 
experts.  Analysis  of  variance  (ANOVA)  was  used  to  determine  the  statistical  significance 
of  the  identified  factors.  ANOVA  results  indicated  that  the  following  factors  have  the 
strongest  impact  on  both  rates  and  times:  alleged  defect  amount,  number  of  issues,  legal 
complexity,  method  of  disposition,  identity  of  prime  contracmr,  product  center,  and 
interest  The  models  developed  explain  73.4%  and  48.5%  of  the  variation  in  sustention 
rates  and  disposition  times,  respectively.  Recommendations  for  improving  sustention  rates 
aixi  disposition  times  based  on  the  research  findings  are  also  included. 
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DEFECTIVE  PRICING:  AN  ANALYSIS  OF  FACTORS  AFFECTING 
SUSTENTION  RATES  AND  DISPOSITION  TIMES 


I.  INTRODUCTION 


Background 

In  response  to  several  1950s  General  Accounting  Office  (GAO)  reports  which 
verified  substantial  overcharging  by  government  contractors.  Congress  passed  the  Troth  in 
Negotiations  Act  (TINA)  in  1962.  The  Act  was  designed  to  place  government  negotiators 
on  equal  footing  with  their  contractor  counterparts  by  requiring  contractors  to  submit  cost 
or  pricing  data  used  in  preparation  of  their  proposals  (IG,  1993:1-1).  Cost  or  pricing  data 
is  defined  as  "all  facts  that,  as  of  the  date  of  agreement  on  the  price  of  a  contract  (or  the 
price  of  a  contract  modification),  a  prudent  buyer  or  seller  would  reasonably  expect  to 
affect  price  negotiations  significantly"  (10  U.S.C.  §2306a(g)).  Failure  to  disclose  current, 
accurate,  and  complete  cost  or  pricing  data  constitutes  defective  pricing  and  entitles  the 
government  to  a  reduction  in  the  contract  price  in  the  amount  of  the  defect 

Despite  the  passage  of  UNA,  defective  pricing  and  the  resultant  overpricing  remain 
significant  and  widespread  (US  GAO,  Dec  1992:7).  During  fiscal  years  1987-1991,  the 
Defense  Contract  Audit  Agency  (DCAA)  identified  $3.67  billion  of  defective  pricing,  with 
approximately  (me  of  every  three  audited  contracts  being  defectively  priced  (US 
GAO/NSIAD-92-184FS,  May  1992:3).  In  addition,  the  alleged  defective  pricing 
amounts,  especially  in  the  area  of  subcontract  defective  pricing,  have  generally  increased 
in  each  successive  year  (US  GAO,  Mar  1991:3). 

Defective  pricing  remains  as  widespread  today  as  it  did  thirty-five  years  ago  because 
of  the  inherent  difficulties  in  interpreting  TINA.  Contractors,  contracting  officers,  the 
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courts  and  boa<tis  of  contract  appeals  continue  to  struggle  with  such  issues  as  the 
definition  of  "cost  or  pricing  data,"  the  required  format  for  data  submission,  the  extent  of 
government  reliance  on  the  defective  data,  and  the  quantification  of  the  price  reduction. 
The  continued  problems  with  defective  pricing  have  necessitated  aggressive  oversight  by 
the  Department  of  Defense  Inspector  General  (DoD  IG),  GAO,  and  Congress.  These 
agencies  have  carefully  monitored  DoD  organizations  to  ensure  effective  handling  of 
defective  pricing  cases.  Their  two  primary  concerns  associated  with  defective  pricing  are 
the  times  required  to  settle  cases  and  the  amounts  of  money  returned  to  the  government. 
Di.SpQSaibOJ[«B^ 

To  increase  the  priority  given  to  defective  pricing  cases,  DoD  directives  classify  a 
defective  pricing  case  as  overage  if  it  has  not  been  settled  within  one  year  from  the  audit 
report  Despite  the  directive,  approximately  50%  of  the  defective  pricing  audits  in  fiscal 
year  1991  were  overage,  with  some  taking  well  over  four  years  to  settle  (US  GAO, 
1993:26).  FurthermcHe,  the  time  period  for  settlements  is  tracked  by  DoD  from  the  latest 
audit  report  rather  tiian  the  initial.  Defective  pricing  settlements  may  involve  several 
supplemental  audit  reports  to  incoqiotate  changes  to  audit  recommendations  as  a  result  of 
contractor  and  contracting  officer  comments.  If  the  time  required  for  settlement  is  tracked 
from  the  initial  audit  report,  the  disposition  times  cited  above  would  increase  significantly. 
In  fact,  the  cases  examined  for  this  research  showed  that  only  27%  were  settled  within  one 
year,  with  the  average  taking  2Vi  years.  In  order  to  capture  the  total  time  spent  resolving 
a  defective  pricing  case,  this  thesis  uses  the  lengthier  definition:  disposition  time  is  the 
time  between  the  issuance  of  the  initial  defective  pricing  audit  report  and  the  contract 
modification  which  implements  the  defective  pricing  settlement  with  the  contractor.  Due 
to  DoD's  poOT  historical  record  of  meeting  established  disposition  timelines  and  the 
heightened  attention  by  oversight  organizations,  this  research  examines  the  underlying 
factors  that  inhibit  or  foster  timely  disposition  of  defective  pricing  cases. 
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Sustention  Rates 

DoD  agencies  are  measured  not  only  on  their  ability  to  settle  cases  in  a  timely 
manner,  but  also  on  their  ability  to  sustain  the  auditor's  recommended  price  adjustment 
The  Deputy  Inspector  General's  testimony  before  the  Senate  Committee  on  Governmental 
Affairs  expressed  his  concern  over  the  continuing  decline  in  DoD's  sustention  rates 
(Senate,  1992:15).  Sustention  rate  is  the  percentage  of  the  audit's  final  recommended 
price  adjustment  (RPA)  that  is  sustained  during  negotiations  or  litigation.  DoD's 
sustention  rates  have  steadily  declined  from  49%  in  fiscal  year  1988  to  an  all-time  low  of 
40%  in  fiscal  year  1992  (US  GAO,  1993:12).  Recommended  price  adjustments  were  not 
sustained  due  to  a  number  of  general  factors  including  legal  opinions  and  litigation 
decisions,  global  settlements,  non-reliance  of  the  government  on  the  defective  data, 
inappiopiiate  contracting  officer  determinations,  contracting  officer  disagreement  with 
auditor’s  recommendaticKis,  audit  enors,  lack  of  management  commitment,  and  lack  of 
experience  and  expertise  of  contracting  officers  (IG,  1990:11;  US  GAO,  1993:12-18). 
Due  to  the  high  visibility  of  declining  sustention  rates,  the  DoD  IG  has  recommended  an 
analysis  of  the  major  factors  affecting  sustention  performance  (IG,  1990:15).  This  thesis 
examines  the  underlying  factors  that  inhibit  cn*  foster  successful  recovery  of  defective 
pricing  funds. 

Specific  Problem 

With  the  exception  of  a  small  number  of  GAO  and  DoD  IG  studies,  no  research  has 
been  conducted  to  identify  or  validate  factors  that  significantly  impact  sustention  rates  or 
disposition  times.  This  thesis  attempts  to  fill  that  void.  The  purpose  of  this  research  is  to 
identify  factevs  which  sigiuficantly  affect  sustention  rates  and  disposition  times  and 
determine  if  valid  models  can  be  developed  to  predict  the  relationships  between  these 
factOTS  and  sustention  rates  and  disposition  times.  With  last  year's  backlog  of  $1.8  billion 
in  defective  pricing  cases  (US  GAO,  1993:28)  and  continuing  manpower  reductions,  a 
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better  understanding  of  the  time  required  to  resolve  cases  and  the  expected  return  on 
effort  is  needed. 

Research  Objectives 

This  research  attempts  to  provide  a  better  understanding  of  the  underlying  factors 
inhibiting  or  fostering  timely  and  successful  recoupment  of  defective  pricing  funds.  More 
specifically,  the  research  objectives  are  to: 

(1)  Identify  factors  that  are  presumed  to  affect  sustention  rates  of  defective  pricing 
cases. 

(2)  Identify  factors  that  are  presumed  to  affect  disposition  times  of  defective  pricing 
cases. 

(3)  Determine  which  factors  identified  in  Objectives  1  and  2  explain  the  greatest 
variation  in  sustention  rates  and  disposition  times. 

(4)  Develop  and  validate  a  model  to  predict  sustention  rates  using  factors  identified 
in  Objective  3. 

(5)  Develop  and  validate  a  model  to  predict  disposition  times  using  factors  identified 
in  Objective  3. 

Sfiscs 

This  research  attempted  to  analyze  all  defective  pricing  cases  resolved  between  1 
March  1991  and  31  March  1994  by  three  of  the  Air  Force  Materiel  Command  (AFMC) 
product  centers:  Aeronautical  Systems  Center  (ASC)  at  Wright-Patterson  AFB,  Space  and 
Missile  Systems  Center  (SMC)  at  Los  Angeles  AFB,  and  Electronic  Systems  Center 
(ESC)  at  Hanscom  AFB.  In  total,  118  cases,  representing  approximately  half  of  the  total 
AFMC  defective  pricing  case  load  during  that  three  year  time  period,  were  analyzed. 
Overview  of  Subsequent  Chapters 

Chapter  2  reviews  existing  literature  in  the  area  of  defective  pricing.  It  explores 
current  interpretative  issues  being  addressed  in  the  courts  and  boards,  procedural  aspects 
concerning  die  disposition  of  defective  pricing  cases,  current  policy  issues,  and  identifies 
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some  potential  causes  of  low  sustention  rates  and  lengthy  disposition  times.  From  this 
review,  an  initial  set  of  factors  presumed  to  affect  sustention  rates  and  disposition  times 
was  generated. 

Chapter  3  describes  the  methodology  used  in  the  research.  It  describes  the 
interview  process  used  to  screen  the  set  of  factors  identified  in  the  literature  review.  It 
also  describes  the  data  collected  to  represent  the  factors  validated  by  experts  interviewed 
and  the  analysis  of  variance  techniques  used  to  determine  which  of  those  factors  explain 
the  greatest  variation  in  sustention  rates  and  disposition  times.  Finally,  the  multiple  linear 
regression  procedures  employed  in  model  building  and  evaluation  are  described. 

Chapter  4  presents  the  results  of  the  research.  It  first  describes  the  results  of  the 
interviews  conducted  with  experts  in  defective  pricing.  Analysis  of  variance  results  are 
then  presented  which  identify  the  factors  most  associated  with  sustention  rates  and 
disposition  times.  Finally,  multiple  linear  regression  nnodels  which  predict  sustention  rates 
and  disposition  times  are  presented. 

Chapter  S  summarizes  the  thesis  research,  discusses  the  factors  which  provide  the 
best  overall  explanation  of  sustention  rates  and  disposition  times,  provides  general 
conclusions  and  findings,  recommends  areas  for  improvement,  and  suggests  ideas  for 
future  research. 


5 


n.  LITERATURE  REVffiW 


Overview 

This  chapter  provides  an  overview  of  defective  pricing  and  discusses  the  legal, 
procedural,  and  policy  issues  which  contribute  to  low  sustention  rates  and  lengthy 
disposition  tunes.  The  chapter  begins  with  a  historical  look  at  the  Truth  in  Negotiations 
Act  and  examines  the  specific  elements  that  the  government  must  prove  to  establish 
defective  pricing.  The  board  and  court  cases  simunarized  in  this  section  highlight  the 
difficulties  in  enfcnrcing  TINA.  Next,  the  process  of  detecting  and  resolving  defective 
{Vicing  cases,  from  initial  audit  through  final  disposition,  is  presented.  Critical  policy 
issues  and  a  summary  of  past  research  are  discussed  last  The  chapter  concludes  with  a 
listing  of  specific  factors,  derived  fiom  this  literature  review,  that  are  presumed  to  affect 
sustention  rates  and  disposition  times.  This  set  of  facters  foam  the  foundation  from 
which  the  remaining  research  is  conducted. 

Truth  In  Negotiations  Act  (TINA) 

Before  TINA  was  enacted,  government  contracting  officers  relied  on  the  expertise 
of  outside  consultants  and  data  voluntarily  submitted  by  the  contractor  to  determine  if  a 
contractcv's  price  was  fair  and  reasonable.  Needless  to  say,  contractors  had  little  incentive 
to  submit  cost  or  pricing  data  that  would  assist  the  government  in  its  negotiations 
(Solimine,  1993).  This  environment  created  a  perception  that  negotiations  lacking 
adequate  price  competition  presented  opportunities  for  unreasonable  prices  and  excessive 
profits  (Shirk  and  Greenberg,  1984:184).  Recognizing  the  government's  vulnerability, 
legislative  changes  designed  to  correct  this  problem  began  in  19S9  and  culminated  with 
the  passage  of  the  Truth  in  Negotiations  Act  in  1962. 
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UNA  was  enacted  to  protect  the  government  against  inflated  prices  by  requiring 
contractors  and  subcontractors  to  submit  cost  or  pricing  data  for  their  proposed  prices 
exceeding  $500,000  and  certify  that  the  data  are  current,  accurate,  and  complete. 
Submission  of  certified  cost  or  pricing  data  is  required  unless  the  contracting  officer 
determines  that  one  of  four  exceptions  applies.  The  four  exceptions  are:  (1)  Contract 
price  is  based  on  adequate  price  competition;  (2)  Price  is  based  on  established  catalog  or 
marlKt  prices  of  commercial  items  sold  in  substantial  quantities  to  the  general  public;  (3) 
Price  is  set  by  law  or  regulation;  or  (4)  Data  submission  is  waived  by  the  Agency  Head  in 
exceptional  cases  (10  U.S.C.  §2306a(b)).  If  none  of  the  exceptions  apply  and  defective 
data  are  found,  TINA  provides  a  contractual  avenue  for  recouping  overpayments  without 
prolonged  litigation  (Boyd,  1993:4). 

The  attitudes  and  perceptions  regarding  TINA  have  changed  dramatically  over  the 
last  thirty  years.  The  period  fix)m  1962  until  the  early  1980s  was  an  "age  of  innocence," 
when  audit  issues  encountered  were  simple  and  case  law  was  generally  predictable.  Next 
came  the  age  of  pricing  "horror  scandals."  Public  outrage  of  overpriced  spares  in  the  early 
1980s  led  to  the  creation  of  a  fraud,  waste,  and  abuse  hotline  and  an  increased  emphasis 
on  the  deterrence  and  punishment  of  overpricing  on  government  contracts.  While  these 
horror  stories  were  generally  not  related  to  defective  pricing,  defective  pricing  nonetheless 
received  heightened  attention.  In  the  mid-1980s,  both  the  number  of  DCAA  personnel 
committed  tt>  defective  pricing  and  the  sophistication  of  their  auditing  increased.  This 
stricter  enforcement  of  TINA  led  to  the  age  of  "active  resistance"  by  contractors  which 
continues  to  date  (JAG  School,  1992:4).  This  environment  also  led  to  the  passage  of 
several  significant  an^ndments  to  TINA,  a  law  which  had  remained  virtually  unchanged 
for  twenty-five  years. 
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In  an  effort  to  encourage  faster  resolution  of  defective  pricing  cases  and  put  "teeth" 
into  the  consequences,  the  1985  amendments  to  TINA  required  the  contractor  to  pay 
interest  on  the  amount  of  overpayment.  It  also  provided  for  a  penalty  assessment,  in 
addition  to  the  interest,  if  defective  data  were  knowingly  submitted  (Sec.  934  of  DoD 
Authorization  Act  for  FY  1986,  P.L.  99-145). 

The  following  year,  TINA  was  again  amended  due  to  renewed  concern  over 
apparent  overcharging  (IG,  1993:1-2).  These  amendments  provided  the  first  statutory 
definition  of  cost  or  pricing  data  and  limited  the  circumstances  in  which  a  contractor  can 
assert  an  offset  against  a  defective  pricing  liability.  They  expressly  eliminated  certain 
contractor  defenses  that  had  historically  been  allowed  by  the  courts.  Assertions  that  the 
defective  data  did  not  cause  an  increase  in  price  because  the  contractor  was  in  a  superior 
bargaining  position  or  was  the  sole  source  can  no  longer  be  used  in  defense  of  a  defective 
pricing  allegation.  Similarly,  defenses  that  the  contracting  officer  should  have  known  that 
the  data  were  defective,  that  the  contractor  failed  to  execute  the  required  certification  of 
cost  or  pricing  data  or  that  the  contract  was  awarded  on  a  bottom-line  price  agreement 
were  also  eliminated  (Sec.  952  of  DoD  Authorization  Act  for  FY  1987,  P.L.  99-500). 
Phase  1:  Burden  of  Proof 

Resolving  a  defective  pricing  allegation  is  a  two-part  process.  The  first  phase  is  the 
determination  that  defective  pricing  occurred  and  the  government  is  entitled  to  a  reduction 
in  the  contract  price.  The  second  phase  is  the  quantification  of  that  reduction.  To 
establish  entitlement  to  a  reduction,  the  Government  bears  the  burden  of  proof  and  must 
address  the  following  proof  elements  (DCAA,  1993:1401): 

(1)  The  data  in  question  meets  the  definition  of  cost  or  pricing  data. 

(2)  Current,  accurate,  and  complete  data  existed  and  were  reasonably  available  to 


the  contractor  before  the  agreement  on  price. 


(3)  Current,  accurate,  and  complete  data  were  not  submitted  or  disclosed  to  the 
government  negotiator. 

(4)  ITie  government  relied  on  the  defective  data  in  negotiating  with  the  contractor. 

(5)  The  government's  reliance  on  the  defective  data  caused  an  increase  in  the 
contract  price. 

Since  refutation  of  any  of  these  proof  elements  constitutes  a  valid  contractor 
defense,  each  element  has  been  the  subject  of  significant  controversy  in  the  boards  and 
courts.  The  following  summary  of  the  case  law  for  each  element  is  presented  to  highlight 
the  inherent  complexities  in  resolving  defective  pricing  cases  in  a  timely  and  effective 
manner. 

Proof  Element  1 :  Defective  Data  are  Cost  or  Pricing  Data.  The  definition  of  cost  or 
pricing  data  consists  of  three  basic  elements:  (1)  facts  (but  not  judgments),  (2)  significant 
^ect  on  price  negotiations,  and  (3)  existing  at  the  date  of  agreement  on  price.  (Morrison 
and  Ebert,  1989:6)  Contractors,  contracting  officers,  the  Armed  Services  Board  of 
(Contract  Appeals  (ASBCA)  and  the  courts  continue  to  struggle  with  interpreting  these 
seemingly  simple  terms. 

(1)  Facts  (but  not  Judgments).  In  defining  cost  or  pricing  data,  the  statute 
states  explicitly  that  cost  or  pricing  data,  "...does  not  include  information  that  is 
judgmental,  but  does  include  the  factual  information  from  which  a  judgment  was  derived" 
(10  U.S.C.  §2306a).  The  distinction  between  fact  and  judgment  is  based  on  the  premise 
that  facts  can  be  verified  and  judgments  cannot.  In  T^mhert  Engineering  Co..  ASBCA 
No.  13338, 69-1  BCA,  the  board  held  that  verifiable  labor  hour  estimates  derived  from 
hismrical  data  are  cost  or  pricing  data  (Boyd,  1993:47).  In  contrast,  the  board  held  in  Ei 
Systems.  Inc..  ASBCA  No.  17557, 74-2  BCA  10,782,  aff d  on  recon.  74-2  BCA  943,  that 
the  profit  plan  dealing  with  prospects  for  future  business  and  the  resulting  effect  of 
overhead  was  speculative  in  nature  and  not  cost  or  pricing  data  (IG,  1993:6-15). 
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Management  decisions  based  on  judgmental  information  must  be  disclosed,  however,  even 
if  the  decision  has  not  been  implemented  (JAG  School,  1992:21).  Despite  board 
decisions,  the  determination  whether  a  judgment,  projection,  or  estimate  is  defined  as  cost 
or  pricing  data  is  a  question  of  fact  that  must  be  determined  on  a  case  by  case  basis  (Boyd, 
1993:48). 

(2)  Significant  Effect  on  Price  Negotiations.  The  issue  of  significance  is 
the  degree  to  which  "prudent  buyers  and  sellers"  would  expect  the  data  in  question  to 
affect  the  contract  price.  Most  court  cases  on  this  issue  involved  non-reported  vendor 
quotations.  "Cost  and  pricing  data  is  made  up  of  costs  which  may  (emphasis  added)  or 
will  make  up  part  of  the  total  cost  of  a  contract..."  (189  Ct.  Q.  at  89-90).  The 
unanswered  question  is  which  vendor  quotes  may  become  a  part  of  the  total  cost  of  the 
contract?  The  possibility  of  selecting  a  supplier  makes  that  quote  cost  or  pricing  data, 
even  if  the  proposal  is  not  based  on  the  quote  or  the  quote  is  nonresponsive  as  initially 
submitted.  But  if  a  quote  is  so  outrageous  that  a  prudent  buyer  or  seller  would  not 
consider  it,  case  law  has  indicated  that  the  quote  is  not  cost  or  pricing  data  (Lovitky, 
Winter  1987:82;  Boyd,  1993:55), 

(3)  Existing  at  the  Date  of  Agreement  on  Price.  A  fact  is  considered  cost 
or  pricing  data  only  if  it  is  significant  "as  of  the  date  of  agreement  on  the  price"  (10  U.S.C. 
§2306(a)(g)).  Therefore,  facts  arising  after  that  date  are  not  cost  or  pricing  data,  even  if 
they  are  available  prior  to  certification  or  contract  award.  This  determinaticm  has  been 
ctmsistentiy  upheld  in  the  courts. 

Proof  Element  2:  Cost  or  Pricing  Data  Existed  and  Were  Available.  In  most  cases, 
if  the  data  existed  prior  to  price  agreement,  it  will  be  considered  reasonably  available 
(Boyd,  1993:127).  This  is  troublesome  to  contractors  because  the  mere  receipt  of  a 
vendor  quotation  within  an  organization  does  not  mean  that  it  will  instantaneously  be 
provided  to  the  contractor's  negotiators.  Normal  processing  delays  are  inevitable 
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(Lovitky,  Sununer  1987:63).  An  excqjtion  to  the  general  rule  of  assumed  availability 
occurs  when  government  urgency  does  not  allow  the  contractor  time  to  update  its  cost  or 
pricing  data  (LTV  Electrosvstems.  Inc..  ASBCA  No.  16802, 73-BCA  f  9957,  afFdon 
recon  denied.  74-1  BCA  10,380). 

Proof  Element  3:  Cost  or  Pricing  Data  Were  Not  Submitted.  While  TINA  and  its 
implementing  regulations  are  explicit  about  the  requirement  to  submit  cost  or  pricing  data, 
very  little  guidance  is  provided  on  the  specifics  of  that  submission.  What  is  clear  is  that 
the  data  must  be  submitted  to  the  cognizant  contracting  officer  or  a  designated 
representative  of  the  contracting  officer.  In  addition,  making  corporate  records  available 
for  government  inspection  is  not  sufficient  Instead,  disclosure  must  be  made  by  physical 
delivery  of  the  data  or  by  specific  identification  (McDonnell  Douglas  Corporation. 

ASBCA  No.  12786, 69-2  BCA  7897  (1969)). 

Other  areas  of  submission  requirements  are  less  clear.  The  statutory  requirement  is 
to  submit  and  certify  cost  or  pricing  data  and  neither  the  act  nor  the  regulations  require  the 
contractor  to  create,  collect,  arrange,  evaluate,  or  review  cost  or  pricing  data  (Doke, 
1989:24).  In  American  Machine  &  Foundry,  the  Board  held  that  contractors  should  not 
have  to  "lead  the  government  by  the  hand,  forcing  it  to  analyze  relevant  data,  properly 
submitted,  but  which  the  Government  chooses  or  happens  to  ignore"  (American  Machine 
&  Foundry  Co..  ASBCA  No.  15037, 74-1  BCA  ^  10,409).  In  another  case,  the  court  held 
that  the  contractor  must  explain  the  significance  of  the  data  if  it  is  not  clear  to  the 
Government  pricing  personnel  (Svlvania  Electric  Products.  Inc,  v.  U.S.  202  Ct.  Cl.  16, 

479  F2d  1342  (1973)).  While  the  case  law  is  inconsistent,  it  does  appear  that  raw  data  is 
often  insufficient  Instead,  the  data  must  be  organized  and  indexed  to  give  it  meaning  and 
relevance  (Lovitky,  Summer  1987:62). 

Proof  Element  4:  Government  Relied  on  Defective  Data.  The  government’s  burden 
of  pioof  for  this  element  has  been  substantially  reduced  by  regulation  and  Board  decisions 


11 


(Boyd,  1993:127-128).  The  appeal  of  American  Bosch  Arma  Corp..  ASBCA  10,305,  65- 
2  BCA  5280,  created  a  refutable  presumption  that,  with  no  evidence  to  the  contrary,  the 
defective  data  caused  dollar-for-dollar  "natural  and  probable  consequences."  To  cause 
these  "natural  and  probable  consequences,"  it  must  be  assumed  that  the  data  were  relied 
upon.  This  presumption  effectively  shifts  the  burden  of  proof  to  the  contractor  who  may 
rebut  by  proving  that  the  data  were,  in  fact,  not  relied  upon. 

Proof  Element  5:  Reliance  on  Defective  Data  Caused  an  Increase  in  Contract  Price. 
Logically,  if  the  omitted  data  did  not  affect  the  contract  price,  the  government  should  not 
be  entitled  to  a  price  reduction.  As  in  the  issue  of  reliance,  the  need  for  the  government  to 
prove  that  the  defective  data  caused  an  increase  in  price  has  been  all  but  eliminated  by  the 
refutable  presumption  of  "natural  and  probable  consequences."  Therefore,  the  issues  of 
reliance  and  causation  have  essentially  been  merged  and  once  the  submission  of  defective 
data  is  established,  the  contract  price  will  be  reduced  unless  the  contractor  can  rebut  that 
presumption. 

Phase  2i_Ouantifying  the  Defect  Amount. 

After  the  government  proves  that  defective  pricing  has,  in  fact,  occurred  (Phase  1), 
the  case  must  be  negotiated  to  determine  the  amount  by  which  the  contract  price  shall  be 
reduced  (Phase  2).  According  to  Rudland  and  Lovitky,  the  difficulty  in  quantifying  the 
price  reduction  arises  from  having  to  determine  (1)  how  the  defective  data  influenced  the 
final  negotiated  agreement  and  (2)  what  would  have  been  negotiated  if  the  correct  data 
had  been  disclosed  (1992:5;  1988:429).  The  problem  associated  with  the  first 
determination  is  that,  "where  a  lump-sum,  final  contract  price  has  been  reached  following 
total  price  offers  and  counteroffers,  there  is  little  way  of  knowing  what  the  negotiators 
agreed  to  with  respect  to  the  cost  element  in  question,  much  less  how  the  defective  cost 
data  influenced  the  lump-sum  price"  (Rudland,  1992:6).  The  dollar-for-dollar  method, 
where  the  contract  price  is  adjusted  downward  by  the  exact  amount  of  the  defect,  is  the 
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least  speculative,  easiest  to  apply,  and  generally  the  most  accepted  (Lovitky,  1988:427). 
The  second  determination-what  the  negotiators  would  have  done,  had  the  appropriate 
cost  or  pricing  data  been  disclosed— is  even  more  speculative  and  virmally  impossible  to 
conclude.  The  boards  have  contrived  a  variety  of  solutions,  ranging  from  acceptance  of 
the  government's  position  of  what  would  have  happened  (if  deemed  reasonable),  to 
construction  of  agreements  which  in  all  likelihood  would  not  have  been  struck  by  the 
parties  (Rudland,  1992:12). 

A  contractor  can  reduce  its  liability  for  defective  pricing  by  demonstrating 
entitlement  to  an  offset.  This  means  that  defective  data  which  overstated  the  contract 
price  can  be  offset  by  data  which  understated  the  price.  The  offset  must  arise  out  of  the 
same  pricing  action  as  the  government's  claim  for  defective  pricing  and  must  not  have  been 
intentionally  inserted  in  the  original  price  agreement  by  the  contractor.  The  government's 
defenses  to  the  proposed  offset  are  the  same  as  the  contractor's  defenses  for  the  defective 
pricing  allegation  (Morrison  and  Ebert,  1989:14). 

D‘..position  Process 

In  discussing  the  law  governing  defective  pricing,  the  elements  required  to 
substantiate  an  allegation  and  quantification  of  the  defects,  the  necessary  groundwork  to 
understand  defective  pricing  has  been  laid.  This  section  describes  the  process  used  by 
DoD  to  detect  and  resolve  defective  pricing  cases. 

Detection.  Defective  pricing  is  usually  discovered  during  a  postaward  audit  by 
DCAA  and  to  a  lesser  extent,  the  DoD  IG  and  GAO.  During  these  audits,  die  records  and 
data  available  to  the  contractor,  as  of  the  date  of  price  agreement,  are  analyzed  and 
compared  with  the  previously  disclosed  cost  or  pricing  data.  The  auditor  investigates  the 
five  proof  elements  discussed  eariier  to  determine  whether  defective  pricing  has  occurred. 

Only  a  limited  number  of  pricing  actions  are  selected  by  DCAA  for  these  postaward 
audits.  The  selection  is  determined  by  two  factors:  an  annual  assessment  of  the  risk  that 
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major  contractors  pose  for  defective  pricing  and  the  dollar  amount  of  the  contract  action. 
The  selection  plan  initiated  in  fiscal  year  1992  evaluates  the  contractor  on  four  factors:  (1) 
Estimating  System  Deficiencies;  (2)  Accounting  System  Deficiencies;  (3)  Incidence  of 
Defective  Pricing;  and  (4)  Amount  of  Recommended  Price  Adjustments.  If  a  contractor  is 
evaluated  as  high  risk,  DCAA  audits  all  negotiated  fixed  price  contracts  worth  $10  -  $25 
million  (US  GAO,  Jun  1992:3).  The  overall  effect  of  this  evaluation  shows  that  a 
relatively  small  number  of  contractors  each  year  account  for  approximately  80%  of  the 
defective  pricing  (US  GAO,  May  1992:2). 

Resolution.  When  an  audit  report  alleging  defective  pricing  is  completed,  DCAA 
sends  it  to  the  cognizant  contracting  agency.  Department  of  Defense  Directive  7640.2, 
"Policy  for  Follow-up  on  Contract  Audit  Reports,"  provides  the  contract  agency 
procedural  guidance  for  the  disposition  of  audit  reports.  The  following  is  an  outline  of  the 
steps  required  to  resolve  a  defective  pricing  case  (IG,  1993:2-1  to  2-4): 

1.  Contracting  Officer  (CO)  receives  the  initial  audit  report  and  ensures  information 
is  entered  into  the  audit  follow-up  tracking  systertL 

2.  CO  reviews  the  report.  In  virtually  aU  of  the  complex  cases,  DCAA  will  have 
discussed  the  findings  with  the  CO  while  the  audit  was  in  draft  form. 

3.  CO  sends  a  copy  of  the  audit  report  to  the  contractor  and  requests  a  written 
response.  In  most  cases,  DCAA  has  already  provided  the  contractor  with  the  specifics 
during  its  audit 

4.  CO  receives  contractor  response  and  allows  DCAA  to  comment.  DCAA  may  or 
may  not  issue  supplemental  audit  reports  changing  the  recommended  price  adjustment 
(RPA)  based  on  contractor  and  contracting  officer  comments. 

5.  CO  establishes  prenegotiation  objective  after  reviewing  comments. 

6.  CO  conducts  negotiations  with  the  contractor  and  reaches  an  agreement. 
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7.  CO  prepares  necessary  support  documentation,  such  as  the  Price  Negotiation 
Memorandum  (PNM)  and  contract  modification. 

8.  CO  obtains  clearance  reviews,  including  legal  review  if  required. 

9.  CO  obtains  contractor  signature  on  contract  modification  (if  bilateral),  distributes 
the  modification,  and  ensures  that  proper  debt  collection  procedures  are  performed. 
Critical  Policy  Issues 

As  evidenced  fix)m  the  discussions  above,  there  are  many  legal  and  quantification 
problems  that  must  be  resolved  and  administrative  steps  to  be  accomplished  prior  to  case 
disposition.  There  are  also  several  critical  policy  issues  which  must  be  considered  during 
the  disposition  process.  These  policy  issues  are  critical  because  they  directly  impact  the 
implementation  and  effectiveness  of  TINA  and  the  ability  of  connoting  officers  to 
successfully  resolve  defective  pricing  cases. 

Distinction  Between  Defective  Pricing  and  Fraud.  An  important  misunderstanding 
is  the  distinction  between  defective  pricing  and  fraud.  Defective  pricing  cases  typically 
deal  with  technical  interpretations  of  the  submission  requirements.  For  these  cases,  the 
contractor  has  submitted  substantial  data,  but  inadvertently  omits  a  particular  document. 

In  contrast,  fruud  cases  usually  involve  either  falsification  of  data  or  a  scheme  (e.g., 
patterns  of  systematic  defects)  to  obtain  higher  prices  (Nash  and  Cibinic,  Dec  1990: 180- 
181). 

Despite  rather  clear  guidelines  of  what  constitutes  fraud  by  the  courts  and  boards, 
there  is  a  perception,  especially  among  contractors,  that  all  defective  pricing  cases  are 
referred  for  fraud  investigation.  This  perception  has  been  fueled  by  the  Inspector  General 
teccxnmendation  that  all  defective  pricing  should  be  referred  for  criminal  investigation. 
(Nash  and  Cibinic,  Sep  1990:129).  Contrary  to  contractor  perceptions,  the  vast  majority 
of  defective  pricing  cases  were  not  referred  for  criminal  investigation.  In  fact,  only  0.5% 
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(16)  of  the  defective  pricing  cases  in  flscal  year  1992  were  referred  to  the  Defense 
Criminal  Investigative  Services  (IG,  1993:1-3). 

Given  the  financial  and  technical  complexities  involved  in  pricing  a  government 
contract,  the  potential  for  inadvertent  defective  pricing  is  always  present  (Shirk  and 
Greenberg,  1984:322).  Referring  all  defective  pricing  cases  for  firaud  investigation  mixes 
together  the  intentional  and  imintentional  violations.  This  mixing  imposes  absurd  litigation 
and  financial  risks  on  the  inadvertent  offender  and  substantially  increases  the  government's 
disposition  times  (Joseph  and  others,  1990:151). 

Subcontracts.  Another  critical  issue  is  the  significant  anrount  of  defective  pricing 
currently  being  discovered  in  subcontracts.  In  the  three  year  period  ended  1990,  63%  of 
all  defective  pricing  cases  involved  subcontracts  (US  GAO/NSIAD-92-131,  May  1992:5). 
The  amount  of  defective  pricing,  as  a  percentage  of  contract  value,  was  highest  in  smaller 
dollar  subcontracts  (US  GAO,  Mar  1991:8). 

Additional  concern  in  the  subcontract  area  arises  from  DCAA's  limited  audit 
coverage  of  subcontractor  proposals.  This  limitation  is  due  primarily  to  the  lack  of  a 
complete  and  accurate  account  of  ail  subcontracts.  In  its  1992  review  of  four  major  DoD 
prime  contracts,  valued  at  $1. 1  billion,  GAO  found  that  DCAA  was  not  aware  of  1 86  of 
the  21 1  related  subcontracts.  Those  186  subcontracts  had  a  total  value  of  $189  million 
(US  GAO/NSIAD-92-173,  May  1992:2).  In  addition  to  the  obvious  inability  to  audit 
subcontracts  it  is  not  aware  of,  DCAA  performs  almost  no  audits  of  low  dollar  value 
subcontracts.  As  such,  the  already  appalling  finding  that  63%  of  the  defective  pricing 
cases  are  related  to  subcontracts  may  actually  be  a  gross  understatement  of  what  is  tmly 
occurring. 

Sweep  Data.  A  current  issue  is  the  growing  use  of  contractor  sweeps  to  collect  cost 
or  pricing  data.  A  data  sweep  is  an  update  of  all  cost  or  pricing  data  that  is  submitted 
aftw  agreement  on  price,  but  prior  to  the  submission  of  the  Certificate  of  Current  Cost  or 
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Pricing  Data.  Although  certificates  are  required  from  the  contractor  as  soon  as  practicable 
after  price  agreement  is  reached,  sluggish  submissions,  some  in  excess  of  30  days,  have 
delayed  execution  of  agreements  and  undermined  the  intent  of  the  law.  Contractors 
contend  that  these  extensive  sweeps  are  necessary  to  ensure  that  all  submitted  cost  or 
pricing  data  were  cuirent,  accurate,  and  complete. 

On  7  June  1989,  the  Deputy  Assistant  Secretary  of  Defense  for  Procurement  issued 
a  policy  letter  instructing  the  contracting  officer  to  "require  the  contractor  to  provide  an 
impact  statement  summarizing  the  impact  of  the  additional  data"  provided  as  the  result  of 
a  sweep  (OASD  Ltr).  This  policy  is  controversial  because  there  is  no  provision  for  such 
impact  statements  in  the  law  or  regulations.  An  additional  problem  with  sweep  data  is  the 
government's  direction  to  reduce  the  agreed  upon  price  if  it  was  inflated  because  of  the 
failure  to  submit  the  sweep  data  before  price  agreement;  no  such  allowance  is  made  for 
upward  adjustment  (IG,  1993:3-5).  Nash  and  Cibinic  have  challenged  these  instructions, 
arguing  that  "whether  sweep  data  should  be  used  to  decrease  or  increase  the  agreed  upon 
price  is  a  matter  for  negotiation  and  is  not  a  subject  for  unilateral  CO  determination" 
(1993:84). 

Interest  Payments  and  Penalty  Provisions.  Congress  added  an  interest  provision  to 
TINA  because,  under  the  original  legislation,  contractors  had  little  incentive  to  expedite 
settlements  on  defective  pricing  cases.  The  effectiveness  of  this  new  incentive  tool  is 
questionable,  however,  because  DoD  has  not  fully  utilized  the  provision.  Contracting 
officers,  acting  without  specific  authority,  have  not  always  charged  the  full  amount,  if  any, 
of  the  interest  due  on  overpayments  (US  GAO,  1993:20). 

A  penalty  provision,  which  doubles  the  amount  owed  by  the  contractor,  was  also 
added.  The  penalty  is  imposed  when  defective  data  are  knowingly  submitted,  thus 
recognizing  the  distinction  between  intentional  and  unintentional  defective  pricing.  Under 
the  original  provision,  the  chance  of  having  to  pay  back  a  deliberate  overpricing  amount. 
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albeit  with  interest,  is  not  very  risky,  and  may  not  be  an  effective  deterrent  for  the 
intentional  offender.  This  provision  was  designed  to  solve  that  problem,  while  not 
imposing  undue  hardship  on  unintentional  defective  pricers.  As  with  the  interest 
provision,  however,  the  penalty  provision  has  not  been  well  implemented.  The  regulations 
provide  no  assistance  on  assessing  the  penalty,  and  DoD  officials  know  of  no  instances 
where  the  penalty  has  been  assessed. 

Factors  Suggested  bv  Oversight  OrgartizatitHis 

The  impact  of  legislative  and  policy  issues,  as  well  as  the  cumbersome  process  to 
detect  and  resolve  a  defective  pricing  case,  have  taken  a  toll  on  the  government's  ability  to 
recoup  the  auditor's  recommended  price  adjustment  and  settle  cases  within  the  one  year 
guideline.  Additional  factors  contributing  to  low  sustention  rates  and  lengthy  disposition 
times  have  been  identified  by  GAO  and  DoD  IG  through  their  investigations  and  are 
discussed  below. 

Low  Sustention  Rates.  For  defective  pricing  cases  settled  in  fiscal  year  1992,  DoD 
reported  that  40%  of  the  $239  million  in  recommended  price  adjustments  was  sustained. 
This  latest  rate  reflects  a  drc^  of  9%  over  the  previous  five  years  (US  GAO,  1993: 12). 
The  GAO  identified  the  following  causes  of  poor  sustention  rates  (US  GAO,  1993:13-18): 

1.  Audit  report  had  errors  or  contained  facts  that  were  inconclusive. 

2.  Government  had  insufficient  evidence  to  counter  contractor's  claim  regarding 
data  availability  due  to  deficiencies  in  record  keeping  and  government  Price  Negotiation 
Memoranda  (PNM). 

3.  Contracting  officers  did  not  rely  on  the  defective  data. 

4.  Inappropriate  contracting  officer  determinations  reduced  or  elirrunated  the  price 
adjustments. 

5.  Negotiations  reduced  the  price  adjustment  to  avoid  administrative  costs  and 
litigation. 
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6.  Contracting  officers  accepted  contractor  offsets,  even  though  DCAA  did  not 
consider  them  valid. 

The  IG  identified  the  following  reasons  for  low  sustention  rates  (IG,  1990:1 1-13): 

1.  Contracting  officers  said  that  they  did  not  rely  on  defective  data  when 
negotiating  the  contract  price. 

2.  Contracting  officers  accepted  data  provided  by  contractors  during  negotiations. 

3.  Contract  audit  reports  contained  inaccurate  or  outdated  information. 

4.  Legal  counsel  advised  against  sustention  because  the  audit  issues  were  not 
supportable. 

5.  Contracting  officers  disagreed  with  the  auditor's  position. 

6.  Contracting  officers  agreed  to  bottom-line  settlements  because  agreement  on 
individual  cost  elements  could  not  be  reached  with  contractors. 

7.  The  sustained  amount  was  part  of  a  global  settlement. 

8.  Contractor  appealed  and  court  or  boaid  decided  issues  in  favor  of  the  contractor. 

9.  Ctmtracting  officers  considered  the  amount  not  sustained  immaterial. 

10.  Events  after  the  audit  report  was  issued,  such  as  finalization  of  rates,  changed  the 
advisability  of  sustention. 

The  IG  provided  two  recommendations  to  improve  sustention  rates.  The  first  was 
to  adhere  to  required  review  and  clearance  procedures  to  ensure  that  contracting  officer 
determinations  are  fully  supported.  The  second,  to  analyze  settlement  data  to  identify 
major  factors  afiFecting  sustention  performance.  On  the  basis  of  the  DoD  IG 
recommendations,  this  research  identifies  and  analyzes  factors  which  affect  sustention 
rates  (IG,  1990:15). 

Lengthy  Disposition  Times.  Because  the  government  has  the  burden  of  proof  in 
establishing  each  of  the  five  elements  for  defective  pricing,  DoD's  disposition  process  is 
inherently  lengthy.  During  settlement,  DoD  is  responsible  for  proving  the  five  basic 
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elements  aiKl  defending  the  amount  to  be  recovered.  In  response,  contractors  must 
provide  data  to  support  their  cost  proposal,  develop  a  point- by-point  rebuttal  to  the 
auditor's  findings,  and  negotiate  or  appeal  the  final  settlement.  Based  on  the  multiple  steps 
that  need  to  be  accomplished  to  complete  a  case,  DoD  settles  only  half  of  its  cases  within 
the  one  year  guideline  (US  GAO,  1993:27).  Other  than  the  cumbersome  process,  lack  of 
experience,  expertise,  priority,  effective  monitoring  controls,  and  management  attention 
are  the  only  other  major  factors  identified  in  the  literature  for  lengthy  disposition  times  of 
defective  pricing  cases  (IG  Ltr,  1993). 

Past  Research 

With  the  exception  of  case  law  analysis  and  DoD  IG  and  GAO  investigations,  little 
research  has  been  done  in  defective  pricing.  This  sectitm  summarizes  what  has  been  done. 

In  1984,  Brown  and  Ellis  analyzed  the  procedures  being  used  by  the  Air  Force 
Systems  and  Logistics  Commands  to  resolve  defective  pricing  cases  and  developed 
management  tools  to  assist  in  the  timely  and  effective  disposition  of  these  cases.  Their 
analysis  revealed  that  the  standard  to  resolve  defective  pricing  cases  was  inadequate  and 
leccHnmended  an  additicxial  45  days.  They  also  recommended  the  establishment  of  a 
single  defective  pricing  OPR,  incentives,  such  as  interest  charges,  to  nx>tivate  timely 
contractor  response,  increased  managerial  emphasis,  and  adq>tic»i  of  expanded  defective 
pricing  milestones  (Brown  and  Ellis,  1984:Ch  5). 

In  her  1988  master's  thesis,  Volpe  demonstrated  that  pursuing  defective  pricing 
allegations  below  $50,000  costs  the  government  more  in  manpower  than  the  funds 
recouped  from  the  contractor.  Therefore,  Volpe  recommended  that  defective  pricing 
cases  under  $50,000  be  pursued  only  on  a  limited  basis.  This  limited  enforcenKnt  would 
loanee  cost  effectiveness  with  the  need  for  an  effective  deterrent  to  defective  pricing, 
regardless  of  dollar  value.  An  ancillary  finding  of  this  research  was  that  the  time  it  took 
to  process  a  case  was  not  related  to  the  allegation  amount.  Instead,  she  found  a 
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conelation  between  processing  time  and  the  contract  value  of  the  action  audited~the 
greater  the  value,  the  longer  it  took  to  settle  the  case  (Volpe,  1988:Ch  5). 


Identification  of  Factors  from  Literature  Review 

Based  on  tte  literature  review  above,  discussions  with  defective  pricing  experts,  and 
posonal  experience,  the  researchers  compiled  a  set  of  factors  for  both  sustention  rates  and 
disposition  times.  Although  suggested  factors  were  generally  specified  for  either  times  or 
rates,  the  researchers  used  the  factors  for  both  times  and  rates,  when  appropriate.  This  set 
of  factors,  along  with  a  brief  definition,  description  of  the  effect  on  rates  and  times,  and 
reference  to  the  source  which  identified  the  factor  is  summarized  in  Table  1. 


TABLE  1 

SUMMARY  OF  SUGGESTED  FACTORS  FOR 
SUSTENTION  RATES  AND  DISPOSITION  TIMES 


Factor 

Definition 

Affect  on 
Rates  &  Times 

Source 

Alleged  Defect 
Amount 

The  auditor's  Recommend  Price 
Adjustment  (RPA) 

Larger  amounts 

Ji  rates,  11  times 

Volpe's  thesis 

Ratio  of  RPA  to 
Audited  Amount 

Proportion  of  the  defect  amount 
to  total  audited  contract  action 

Increased  ratios 

U  rates.  It  times 

Variation 
from  Volpe’s 
thesis 

Cost  Element 

Identifies  which  cost  element  the 
defective  data  pertained  to:  labor, 
material,  indirect  or  other 

Material  & 
labor 

It  rates,  U  times 
as  compared  to 
indirect  costs 

J.A.  Lovitky,  I 
Nat'l  Contract 
Mgt  Journal, 
Winter  1987 

Contractor  Level 

Identifies  which  party,  the  prime 
contractor  or  subcontractor, 
committed  the  defective  pricing. 

Subcontractor 
Unknown  for 
rates.  It  times 

GAO/NSIAD 

-92-131 

Type  of  Contract 
Action 

Indicates  whether  defective 
pricing  occurred  on  an  original 
contract  or  contract  modification 

Original 
contracts 
ti  rates.  It  times 

Defective 
pricing  expert 
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TABLE  1  (CONTD) 


CO 

Disagreement 

withRPA 

Indicates  whether  the  contracting 
officer  disagreed  with  the  RPA 

Disagreement 
w/RPA 
rates.  It  times 

IG  Report  # 
AFU91-1 

Number  of 

Issues 

Number  of  separate  issues 

involved  in  a  single  defective 
pricing  case 

More  issues 

U  rates,  1T  times 

Defective 
pricing  expen 

Complexity 

Indicates  the  complexity  of  the 
legal  Issues  surrounding  the  case 

More  Complex 

U  rates,  1t  times 

GAO/NSIAD 

-94-7 

Offsets 

Indicates  whether  or  not  the 
contractor  proposed  offsets 

Proposed 

offsets 

Unknown  on 
rates.  It  times 

GAO/NSIAD 

-94-7 

Recipient  of 

Funds 

Indicates  whether  the  program 
office  or  the  General  Treasury 
received  the  recovered  funds 

Program  office 
ft  rates,  li  times 

Method  of 
Disposition 

Indicates  how  the  case  was 
settled:  negotiation,  out-of-court 
settlement,  or  litigation. 

Litigation 

U  rates,  tt  times 

IG  Report  # 
AFU91-1 

Fraud 

Investigation 

Indicates  whether  or  not  the  case 
was  investigated  for  fraud. 

Fraud 

investigation 

1t  rates.  It  times 

GAO/NSIAD 

-94-7 

Up-Front  Time 

Number  of  days  between  the 
initial  audit  and  last  siq)plement, 
if  any 

Greater#  of 
days 

It  rates,  tt  times 

GAO/NSIAD 

-94-7 

Disposition  Hme 

Number  of  days  between  initial 
audit  and  modification  issuance 

Longer  times 

U  rates,  N/A  for 
times 

AFMC  audit 

follow-up 

statistics 

Contract  Type 

Indicates  whether  the  case 
involves  a  fixed  price  or  cost 
reimbursement  contract 

Fixed-price 

It  rates.  It  times 

Nash  & 

Cibinic 

Report  Jan 
1991 

Contract  Action 
Audit  Amount 

The  total  dollar  value  of  the 
audited  contract  action 

Larger  amounts. 
It  rates,  tt  times 

Volpe’s 

Thesis 

ContractCH'  Size 

Categorizes  the  contractor  as  a 
large  or  small  business 

Unknown  effect 
on  rates  and 
times 

Researchers 

Contract  Cost 

Indicates  whether  the  contract  is 

Overrun 

Researchers 

Performance _ in  an  overrun  or  underrun  state  U  rates,  It  times 


22 


TABLE  1  (CONTD) 


Identity  of  Prime 
Contractor 

Identifies  the  prime  contractor 
involved  in  the  case 

Contractors 
differ  in  rates 
and  times 

GAO/NSIAD 

-92-183 

Gov't  Negotiator 
Expertise 

Number  of  cases  the  government 
negotiator  had  previously 
resolved 

More  cases 
tt  rates,  times 

Senate  IG 
Testimony, 

4  Jun  92 

Contractor 

Negotiator 

Expertise 

Number  of  cases  the  contractor 
negotiator  had  previously 
resolved 

More  cases 
^  rates, 

unknown  effect 
on  times 

Variation 
from  Senate 

IG  Testimony 

Organizational 

Structure 

Indicates  whether  the  contr^ting 
activity  which  resolved  the  case 
was  centralized  or  decentralized 

Centralized 

1T  rates,  li  times 

Senate  IG 
Testimony, 

4  Jun  92 

Workload 

Captures  the  workload  level  of 
the  gov't  negotiator  at  the  time 
the  case  is  being  resolved 

Higher 

workload 

U  rates.  It  times 

Researchers 

Priority 

Identifies  the  priority  given  to 
cases  relative  to  other  actions 

Lower  priority 

It  rates,  ft  times 

9/93  IG  audit 
review  at  San 
Antonio 

Surninary 

This  chapter  examined  the  current  interpretative  issues  being  addressed  in  the  courts 
and  boards,  discussed  procedural  aspects  of  the  disposition  of  a  defective  pricing  case, 
highlighted  current  policy  issues,  and  examined  the  impact  that  these  issues  have  on 
sustention  rates  and  disposition  times.  Based  on  this  literature  review,  a  list  of  factors 
affecting  sustention  rates  and  disposition  times  was  generated. 

Chapter  3  describes  the  interview  process  used  to  validate  the  list  of  factors 
geiwrated  by  the  literature  review,  the  analysis  of  variance  procedures  that  determined 
which  factors  contributed  significantly  to  explaining  sustention  rates  and  disposition  times, 
and  the  statistical  techniques  employed  in  model  development  and  validation. 
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ffl.  METHODOLOGY 


Overview 

This  chapter  describes  the  three  methods  used  to  achieve  the  research  objectives. 
The  first  two  objectives  were  to  identify  factms  that  affect  sustention  rates  and  disposition 
times.  These  objectives  were  accomplished  by  reviewing  ^plicable  literature  and 
surveying  defective  pricing  experts  through  personal  and  telephone  interviews.  The  third 
olqective  was  to  determine  which  of  the  factors  explain  the  greatest  variation  in  sustention 
rates  and  disposition  times.  This  was  accomplished  through  one-way  analysis  of  variance 
and  multiple  ccxnparison  of  means  tests.  The  last  two  objectives  were  to  develop  and 
validate  two  models  using  the  statistically  significant  factors~one  to  explain  sustention 
rates  and  the  other,  disposition  times.  These  objectives  were  met  using  multiple  linear 
regression  (MLR). 

Identification  of  factors  from  Interviews 

Method  Selection.  On  the  basis  of  a  literature  review,  personal  experience,  and 
discussions  with  defective  pricing  experts  at  Aeronautical  Systems  Center,  a  list  of 
potential  facuxrs  affecting  sustention  rates  and  disposition  times  was  compiled.  This  list  of 
initial  factors  was  presented  in  Chapter  2,  Table  1.  To  validate  this  initial  set  of  factors 
and  identify  important  additional  factors,  it  was  necessary  to  survey  defective  pricing 
experts.  The  goal  was  to  ensure  that  all  variables  affecting  sustention  rates  and  disposition 
times  wav  identified.  To  accomplish  this  goal,  persmial  interviews  and  telephone 
interviews  were  conducted.  Interviews  were  selected  for  two  reasons; 

(1)  Based  on  the  inherent  complexi^  of  defective  pricing  issues,  it  was  considered 
essential  that  a  two-way  dialogue  take  place.  It  was  important  that  the  interviewers  be 
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able  to  answer  questions  about  the  survey,  secure  an  in-depth  and  detailed  response,  and 
probe  for  additional  information  on  potential  factors  (Emory  and  Cooper,  1991:338). 

(2)  Because  of  the  relatively  few  number  of  defective  pricing  experts  (24),  there 
were  sufficient  time  and  resources  to  interview  them  either  in  person  or  by  telephone. 
Individuals  in  the  local  area  were  interviewed  in  person;  those  outside  the  area  were 
interviewed  by  telephone. 

Survey  Development.  The  twenty-four  facmrs  identified  in  Table  1  were 
incorporated  into  a  series  of  closed  response  questions.  The  purpose  of  these  questions 
was  to  have  the  resptxidents  identify  which  of  the  given  factors  they  presumed  affected 
sustention  rates  and  resolution  times,  and  to  what  extent.  The  extent  to  which  each  factor 
affected  rates  and  times  was  measured  on  a  5-point  Likert  scale.  The  scale  ranged  from 
no  effect  (1)  to  very  strong  effect  (5).  After  the  series  of  closed  response  questions,  the 
survey  concluded  with  open  respcmse  questions.  These  questions  allowed  the  respondent 
to  provide  any  addititmal  factors  that  they  presumed  affected  sustention  rates  and/or 
resolution  times,  and  the  degree  of  impact  The  purpose  of  the  open-ended  questions  was 
to  ensure  that  no  worthwhile  factors  were  erroneously  omitted.  The  omission  of  such 
factors  would  have  seriously  diminished  the  researchers'  ability  to  develop  useful  models. 

To  ensure  the  interview  questions  were  clear  and  understandable,  the  draft  survey 
was  reviewed  by  two  AFTT  faculty  members  and  three  defective  pricing  experts  at  ASC. 
Their  comments  were  incorporated  into  the  final  version.  The  survey  is  included  as 
Appendix  A. 

Participant  Selection.  The  selection  process  was  designed  to  incorporate  the 
different  perspectives  of  those  involved  in  the  resolution  and  oversight  of  defective  pricing 
cases.  The  varying  perspectives  are  gained  by  working  in  one  of  three  types  of 
organizaticms;  line,  staff,  or  oversight 
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Line  members,  such  as  contracting  officers  and  negotiators/pricers  who  had  recent 
experience  resolving  defective  pricing  cases,  were  interviewed.  The  defective  pricing  focal 
points  at  each  of  the  three  product  centers  identified  the  line  members  they  considered  to 
be  their  defective  pricing  experts.  Each  suggested  expert  was  then  contacted  to  confirm 
whether  they  had  sufficient  knowledge  and/or  experience  to  competently  answer  the 
survey  questions.  Those  interviewed  in  ASCs  centralized  defective  pricing  organization 
had  between  one  and  five  years  of  experience  resolving  defective  pricing  cases,  with  ten  to 
twenty  cases  being  resolved  per  year.  Those  interviewed  in  SMCs  and  ESC's 
decentralized  organizations  resolved  between  three  and  seven  cases  in  the  last  three  years. 

Insight  into  the  staff  perspective  was  gained  by  interviewing  the  center's  defective 
pricing  focal  points,  DCAA  liaison  auditors,  legal  advisors,  and  Headquarters  AFMC  and 
DCAA  personnel  Defective  pricing  focal  points  disseminate  applicable  policy,  assist  line 
members,  monitor  and  report  resolution  of  defective  pricing  cases,  and  resolve  selected 
cases,  when  necessary.  The  DCAA  liaison  auditors  act  as  liaisons  between  the  DCAA  and 
contracting  personnel  at  the  centers  on  all  audit  matters,  including  defective  pricing.  Two 
lawyers  at  the  Air  Force  Materiel  Conmand  Law  Center,  who  provide  primary  legal 
support  for  the  resolution  of  defective  pricing  cases  for  AFMC,  were  interviewed.  They 
have  over  thirty  years  of  combined  defective  pricing  experience.  The  HQ  staff  participants 
also  had  extensive  and  varied  backgrounds  relating  to  defective  pricing. 

To  incorporate  the  broad  experience  and  perspective  of  an  oversight  organization, 
individuals  from  the  DoD  IG  were  interviewed.  Hie  interviewees  had  reviewed  hundreds 
of  Army,  Navy,  Air  Force,  and  Defense  Logistics  Agency  (DLA)  defective  pricing  cases. 

From  the  various  line,  staff,  and  oversight  organizations,  a  total  of  twenty-four 
interviews  were  ctxiducted.  The  list  of  interviewees  is  included  as  Appendix  B. 

Interview  Technique.  An  advance  copy  of  the  survey  was  sent  to  all  interview 
participants  to  allow  for  adequate  preparation.  The  interviews  were  conducted  with  both 


26 


researchers  present.  To  minimize  disruption  and  possible  confusion  during  the  interview, 
especially  those  over  the  telephone,  the  survey  questions  were  asked  by  only  one 
researcher.  Each  researcher  took  separate  notes  during  the  interview  and  a  comparison 
was  made  immediately  after  each  interview. 

After  ensuring  consistency  in  recording  all  respondents'  comments  and  scoring,  the 
median  was  calculated  for  the  aggregate  factors.  Factors  with  a  median  of  3  (moderate 
effect)  or  greater  were  considered  to  be  significant  and  validated  by  the  interviews.  The 
median  was  chosen  as  the  best  discriminator  because  it  suggests  that  at  least  half  of  the 
respondents  judged  the  effect  to  be  moderate  or  greater. 

PmCQUggtion 

All  of  the  factors  deemed  to  be  significant  during  the  interviews  were  screened  for 
inclusion  in  analysis  of  variance  and  model  building.  To  decrease  the  subjectivity  of 
classificatitMi  by  the  researchers  and  ensure  usefulness  of  the  resulting  models,  only  those 
factors  for  which  data  could  easily  be  collected  were  further  analyzed.  Data  on  sustention 
rates,  disposition  times,  and  the  selected  independent  variables  were  obtained  from  two 
sources:  the  PK  Audit  Reporting  System  (PARS)  and  contract  files. 

PK  Audit  Reporting  System  (PARS).  PARS  is  an  audit  tracking  system  hosted  on 
the  Automated  Management  Information  System  (AMIS)  mainframe.  AFMC's 
Contracting  Automation  and  Data  Systems  Division  queried  the  system  and  provided  a  list 
of  all  defective  pricing  cases  closed  between  1  March  1991  and  31  March  1994.  For  each 
case,  the  audit  report  number  and  date,  contractor,  contracting  center,  cost  questioned, 
cost  sustained,  litigation  number,  resolution  date,  resolution  age,  disposition  date,  and 
disposition  age  were  collected.  Appendix  C  contains  a  complete  listing  of  the  PARS  data 
sheets  used  for  this  research. 

PARS  infOTtnation  was  entered  into  the  system  by  the  defective  pricing  focal  point 
at  the  centers.  To  ensure  accuracy,  many  of  the  input  fields  include  edit  checks  to  prevent 
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entry  of  an  incorrect  value.  Despite  this  precaution,  both  the  GAO  and  DoD  IG  have 
found  numerous  reporting  errors  in  costs  questioned  and  amounts  sustained  (US  GAO, 
1993:29).  As  such,  the  original  contract  files  containing  the  defective  pricing  infonnation 
were  used  to  verify  or  correct  the  data  received  from  PARS,  as  well  as  supplement  data 
not  included  in  the  PARS  system. 

Contract  Files.  The  contracting  centers'  contract  files  were  reviewed  for  pertinent 
facts  and  figures  on  the  factors  being  investigated.  Among  the  documentation  analyzed 
were  audit  reports,  price  negotiation  memoranda,  contract  modifications,  legal  opinions, 
and  contractor  response  letters.  While  not  all  the  contract  files  contained  all  the 
documentation,  there  was  sufficient  information  in  the  majority  of  the  files  to  determine 
what  had  transpired.  When  discrepancies  were  found  between  PARS  and  the  contract  file 
documentation,  the  data  in  the  contract  files  were  deemed  more  reliable  and  used. 

The  contract  file  is  prepared  and  maintained  by  the  contract  negotiator,  in 
accordance  with  applicable  procedures.  The  file  documentation  is  generally  reviewed  for 
accuracy  and  completeness  by  the  contracting  officer  and  legal  counsel,  and,  if  the  dollar 
amount  is  large  enough,  by  a  contracting  committee.  These  internal  reviews  increased  the 
researchers'  confidence  in  the  data's  validity  and  reliability. 

Data  Selection.  This  research  attempted  to  analyze  all  defective  pricing  cases 
resolved  by  AFMC  product  centers  between  1  March  1991  and  31  March  1994.  The 
three  centers  for  which  data  were  gathered  were  Aeronautical  Systems  Center  (ASC)  at 
Wright-Patterson  AFB,  Space  and  Missile  Systems  Center  (SMQ  at  Los  Angeles  AFB, 
and  Electronic  Systems  Center  (ESC)  at  Hanscom  AFB.  The  fourth  product  center. 
Human  Systems  Center  (HSC),  was  not  selected  because,  according  to  PARS,  HSC  did 
not  settle  any  defective  pricing  cases  during  the  specified  time  period.  AFMC's  logistics 
centers  were  also  eliminated  because  of  the  difficulty  in  obtaining  reUable  defective  pricing 
data.  This  difficulty  stems  from  the  recent  Air  Force  Logistics  Command  (AFLC)/Air 
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Force  Systems  Command  (AFSC)  merger  in  July  1992.  HQ  AFSC  had  a  central, 
automated  system  prior  to  the  merger;  HQ  AFLC  did  not.  AFLC's  data  were  described  by 
HQ  AFMCs  defective  pricing  focal  point  as  incomplete  and  inaccurate  prior  to  and  six 
months  after  the  1992  merger.  To  ensure  data  integrity  and  consistency,  this  research 
analyzed  defective  pricing  cases  from  the  AFMC  product  centers  only. 

Data  were  collected  on  all  defective  pricing  cases  contained  in  PARS  for  the  three 
product  centers.  Defective  pricing  data  were  input  into  PARS  beginning  March  1991.  As 
such,  all  defective  pricing  cases  closed  between  1  March  1991  and  31  March  1994  at 
ASC,  SMC,  and  ESC  were  considered  for  this  research. 

It  was  necessary  to  eliminate  certain  defective  pricing  cases  during  the  1991-1994 
time  frame.  Cases  which  had  a  reported  sustention  amount  of  $0  were  eliminated.  These 
were  cases  in  which  the  contracting  activity  determined  that  defective  pricing  had  not 
occurred.  Because  the  contracting  activity  has  the  ultimate  authority  in  determining 
whether  defective  pricing  occurred,  those  cases  with  a  zero  sustention  rate  were  removed 
from  consideration.  The  contract  files  for  an  additional  17  cases  could  not  be  located. 

The  PARS  data  for  these  missing  cases  were  analyzed  and  no  commonalities  were 
identified.  As  such,  it  was  concluded  that  the  missing  files  occurred  at  random  so  their 
absence  does  not  affect  the  research  results.  With  these  exclusions,  data  were  collected  or 
a  total  of  1 18  cases.  The  fitequency  distribution  of  cases  for  each  center  was  as  follows: 
ASC  72%,  SMC  16%,  and  ESC  12%. 

Analysis  of  Variance 

Analysis  of  variance  (ANOVA)  is  used  to  study  the  effects  of  one  or  more 
independent  variables  on  a  dependent  variable.  In  this  research,  ANOVA  techniques  were 
used  to  determine  which  of  the  factors  validated  during  the  interview  process  explain  the 
greatest  variation  in  sustention  rates  and  disposition  times.  The  SAS  statistical  software 
package  was  used  to  perform  the  analyses.  SAS  includes  two  procedures  for  analysis  of 
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variance  -  ANOVA  and  General  Linear  Model  (GLM).  The  ANOVA  procedure  is  limited 
to  situations  involving  equal  cell  sizes.  Because  this  research  involved  observational  data 
that  resulted  in  unequal  cell  sizes,  the  GLM  approach  was  employed. 

ANOVA  Models.  One-way  ANOVAs  were  conducted  for  each  of  the  variables 
validated  during  the  interview  process  to  determine  the  variables'  effects  on  sustention 
rates  and  disposition  times.  The  model  used  for  these  tests  was: 

Yij  =  ^.  +  Xi  +  eij 

whoe  Y  is  the  response  or  dependent  variable  (sustention  rate  or  disposition  time); 

Xj  are  the  factor  levels  of  the  independent  variables; 

t  ^  1,...,  iU, 

m  =  number  of  levels  for  the  independent  variables; 

j  =  1,.,.,  n;  and 

n  =  number  of  observations. 

The  model  results  were  used  for  hypothesis  testing  and  the  coefficient  of  determination 
was  evaluated  to  assess  the  extent  to  which  each  factor  explains  the  variation  in  sustention 
rates  and  disposition  times. 

Hypothesis  Testing.  The  F  test  was  used  for  hypothesis  testing.  The  F  test  is  a 
preliminary  test  which  determines  if  there  is  a  statistically  significant  difference  between 
any  of  the  factor  level  means.  The  hypotheses  tested  in  the  ANOVAs  were: 

H(,:  All  factOT  level  means  are  equal. 

At  least  two  of  the  factor  level  means  are  not  equal. 

If  the  F  test  provided  no  evidence  of  differences  between  the  factor  level  means  at  the  a  = 
.05  level,  it  was  concluded  that  no  relationship  exists  between  the  factor  and  the 
dependent  variable.  If  the  F  test  indicated  that  the  factor  level  means  differ  from  one 
another  and  there  were  only  two  factor  levels,  no  additional  analysis  was  required;  if  tliere 
were  more  than  two  factor  levels,  additional  analysis  was  required  to  determine  the  nature 
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of  the  factor  level  effects.  In  these  cases,  the  Bonferroni  method  of  multiple  comparisons 
was  used  to  determine  which  of  the  factor  level  means  differ  from  one  another.  The 
Bonferroni  method  was  selected  because  of  its  applicability  to  situations  involving  unequal 
cell  sizes  (Neter  and  others,  1985:582).  The  Bonferroni  tests  examined  pairwise 
comparisons  of  the  factor  level  means  at  the  a  =  .05  level. 

Coefficient  of  Determination.  The  coefficient  of  determination,  or  r2,  is  the  ratio  of 
explained  variance  to  total  variance  and  measures  how  well  a  linear  model  fits  a  set  of 
data.  The  value  of  is  always  between  0  and  1,  where  R^  =0  implies  a  complete  lack  of 
fit  of  the  model  to  the  data  and  R2=1  implies  a  perfect  fit,  with  the  model  passing  through 
every  data  point  The  R^  of  each  ANOVA  model  was  reviewed  to  assess  the  amount  of 
variation  in  the  dependent  variable  that  was  explained  by  the  single  independent  variable. 

As.«iumptions.  In  an  analysis  of  variance  model,  it  is  assumed  that  (Neter  and  others, 
1985:524): 

(1)  Each  of  the  probability  distributions  is  normal. 

(2)  Each  probability  distribution  has  the  same  variance. 

(3)  The  observations  for  each  factor  level  are  random  and  independent 

Wilk-Shapiro  rankit  plots  revealed  that  the  distributions  of  the  dependent  variables 

were  not  normal.  However,  the  F-test  used  in  ANOVA  is  robust  against  departures  from 
normality.  As  such,  the  F-test  can  still  be  used  with  little  effect  on  the  level  of  significance 
or  power  of  the  test  (Neter  and  others,  1985:624). 

The  Bartlett  test  was  used  to  evaluate  homogeneity  of  variance.  The  Bartlett  test  is 
sensitive  to  nonnormality,  however,  especially  fear  long-tailed  distributions.  In  such  cases, 
heteroscedasticity  is  detected  too  often.  Andenon  and  McClean  recommend  evaluating 
the  Bartlett  test  at  the  a  =  .001  level  in  these  circumstances  (1974:19).  The  distributions 
of  both  disposition  times  and  sustention  rates  are  nonnormal  with  long  tails  and  several  of 
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the  populations  were  found  to  violate  the  assumption  of  equal  variances  at  the  a  =  .001 
level. 

Heteroscedasticity,  or  unequal  variances,  can  be  corrected  through  transformation 
of  the  dependent  variable  (Anderson  and  McLean,  1974:17)  or  weighted  least  squares. 
Attempts  at  transforming  the  data  through  use  of  ArcSin,  logarithmic,  and  square  root 
functitms  were  unsuccessful.  Therefore,  the  weighted  least  squares  approach  was  used 
with  a  weighting  of  1  over  the  variance  squared  (o^).  Use  of  this  weighting  serves  to 
equalize  all  variances,  thereby  satisfying  the  assumption  necessary  to  ensure  validity  of  the 
statistical  tests  (Neter  and  others,  1985:262). 

In  this  research,  all  of  the  defective  pricing  cases  resolved  in  a  three-year  time-frame 
were  analyzed.  Given  the  nature  of  the  data,  it  is  presumed  that  the  third  assumption,  that 
observations  for  each  factor  level  are  random  and  independent,  is  met 
Model  Building 

Having  identified  a  set  of  factors  for  sustention  rates  and  disposition  times  through 
the  interview  process,  the  factors  were  further  screened  through  analysis  of  variance.  The 
factors  having  means  significantly  different  firom  one  another  at  the  95%  confidence  level 
became  die  initial  set  of  independent  variables  considered  for  model  inclusion.  Data  on 
the  variables  were  entered  in  Statistix  4.0,  a  statistical  computer  program  designed  to 
perform  multiple  linear  regression. 

Multiple  Linear  Regres.sion  t'MI.RL  The  goal  of  MLR  is  to  build  a  model,  using  the 
best  subset  of  independent  variables,  to  predict  values  of  the  dependent  variable.  The 
MLR  models  for  sustention  rates  and  disposition  times  will  be  in  the  following  format: 

Y  =  Po  +  P2X2  +— +  Pk^k  +  £ 
where  Y  is  the  response  or  dependent  variable; 

X],  X2,...,  X|(  are  predictor  or  independent  variables; 
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pQ.  Pi»— .  Pk  ^  unknown  model  parameters  called  regression  coefficients; 

k  is  the  number  of  independent  variables  included  in  the  model;  and 
e  is  the  random  error  term. 

The  following  four  steps  were  taken  to  build  the  MLR  models:  Step  1,  best  subset 
selection;  Step  2,  verification  of  model  assumptions;  Step  3,  model  determination  and 
evaluation;  and  Step  4,  model  validation. 

Step  1:  Best  Subset  Selection.  For  this  thesis,  the  goal  is  not  only  predictive 
accuracy,  but  also  selection  of  independent  variables  that  facilitate  model  use. 
Accomplishing  these  goals  entails:  (1)  selecting  predictor  variables  for  which  data  are 
readily  available  and  (2)  minimizing  the  number  of  predictor  variables  in  the  nxxlels.  The 
former  objective  was  accomplished  through  a  screening  of  the  interview  results  prior  to 
data  collection,  and  the  latter  was  achieved  through  the  application  of  Miller's  Method. 
Miller's  Method  was  chosen,  rather  than  other  subset  selection  techniques,  as  it  guards 
against  bringing  random  predictors  into  the  model.  Research  recently  conducted  by 
Woollaid  showed  that  Miller's  Method  was  the  best  technique,  when  compared  to 
Minimum  MSE,  Minimum  Sp,  Minimum  Cp,  for  selecting  models  from  a  variable  pool 
containing  extraneous  variables  (1993:70). 

Miller's  method  augments  the  set  of  predictor  variables  with  an  equal  number  of 
dummy  variables  whose  values  are  random  numbers.  Miller's  Method  then  applies 
forward  stepwise  regression  and  proceeds  until  the  frrst  dummy  variable  is  selected  for 
inclusion  in  the  model.  Once  this  occurs,  the  regression  stops  and  oiily  those  predictor 
variables  selected  before  the  entrance  of  the  dummy  variable  are  considered  for  inclusion 
in  the  model.  The  rationale,  of  course,  is  that  any  predictor  variable  selected  after  a 
dummy  variable  must  have  less  significance  than  that  random,  dummy  variable.  Thus,  all 
predictor  variables  selected  after  should  be  discarded  as  insignificant  (Miller,  1984:395). 
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Miller’s  Method  was  implemented  in  Statistix  4.0  using  the  Stepwise  Regression 
program  with  forward  selection.  The  F-to-enter  threshold  criteria  was  set  to  zero  to 
ensure  that  all  variables,  including  the  random  ones,  would  be  admitted  into  the  models. 
Miller's  method  was  repeated  ten  times  for  each  model  to  verify  the  selection  of  predictor 
variables. 

Step  2:  Verificarion  of  Model  Assumptions.  The  assumptions  made  when 
conducting  a  MLR  analysis  ate  that  the  error  terms  are  independent  and,  for  any  given  set 
of  X  values,  e  has  a  normal  probability  distribution  with  a  mean  of  zero  and  constant 
variance  (McClave  and  Bensen,  1991:526-527).  Gross  violations  of  these  assumptions 
may  yield  an  unstable  model.  The  following  methods  were  used  to  check  the  validity  of 
diese  assumptions: 

Residual  Analysis.  Residual  analysis  involves  careful  inspection  of  the 
differences  between  the  observed  and  predicted  values  of  the  dependent  variable  after  a 
prediction  equation  is  fit  to  the  data.  Residual  analysis  can  detect  trends  and  extreme 
measurements,  such  as  outliers,  and  identify  assumption  violations,  such  as  changes  in  the 
error  variance  (heteroscedasticity)  and  non-normal  error  distributions  (Gunst  and  Mason, 
1980:220). 

Heteroscedasricitv.  Residual  plots  were  utilized  to  detect 
heteroscedasticity.  Error  variances  which  are  equal  over  the  range  of  independent 
variables  will  show  residuals  randomly  placed  above  and  below  the  zero  line.  Deviations 
from  equal  variance  are  portrayed  in  many  forms,  such  as  an  expanding  cone  or  football 
shape  of  the  residual  plots.  Unequal  variances  are  normally  stabilized  through  a 
transformation  of  the  dependent  variable. 

Normal  Distribution  of  Residuals.  To  check  that  the  random  errors  are 
normally  distributed  with  mean  of  zero,  a  Wilk-Shapiro/Rankit  Plot  was  accomplished. 
The  straighter  the  line  of  rankit  plots  or  the  higher  the  W  Statistic,  the  more  normal  the 
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distribution.  For  purposes  of  this  research,  normality  is  assumed  if  the  rankit  plot  appears 
straight  and  the  W  Statistic  exceeds  0,90. 


Multicollineaiitv.  MulticollineaiiQr  refers  to  the  situation  in  which  some  or  all 
of  the  independent  variables  are  interrelated.  When  highly  correlated  independent 
variables  are  present  in  a  regression  model,  the  results  are  often  confusing.  Indicators  of 
multicollinearity  include:  a  large  value  and  F-Statistic,  but  small  individual  t- values; 
large  changes  in  the  estimated  coefficients  when  a  variable  is  added  or  deleted;  large 
changes  in  the  coefficients  when  a  data  point  is  altered  or  dropped;  large  standard  errors 
for  coefficients  that  are  expected  to  be  important;  and  large  variance  inflation  factors 
(Chatteijee  and  Price,  1977:144-156).  These  indicators  of  multicollinearity  were 
evaluated  using  standard  regression  computations. 

Step  3:  Model  Determination  and  Evaluation.  To  test  the  usefulness  of  the  models, 
a  global  test,  one  that  encompasses  all  |3  parameters,  is  required.  Statistical  measures  are 
also  necessary  to  evaluate  how  well  a  linear  model  fits  the  data.  The  methods  which  were 
selected  and  performed  are  described  below. 

F-Test.  An  analysis  of  variance  F-Test  is  conducted  after  a  multiple  linear 
model  is  developed  and  all  assumptions  are  satisfied.  The  F  statistic  is  the  ratio  of  the 
explained  variance  over  the  unexplained  variance.  The  F-Test  indicates  whether  a 
significant  linear  relationship  exists  between  the  dependent  variable  and  the  combined 
effects  of  the  independent  variables.  The  Global  F-Test  is  usually  regarded  as  a  test  that 
the  model  must  pass  to  merit  further  consideration  (McClave  and  Bensen,  1991:543). 

Coefficient  of  Determination.  The  coefficient  of  determination,  R2,  represents 
the  prcqxntion  of  the  total  variability  explained  by  the  linear  relationship  and  was  used  to 
determine  how  well  the  linear  models  tit  the  data.  R^  was  selected  because  it  is  regarded 
as  the  natural  measure  of  the  uncertainty  of  predicting  the  dependent  variable.  In  general, 
the  larger  the  value  of  R^,  the  better  the  model  predicts  the  dependent  variable. 
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Step  4:  Model  Validation.  Model  validation  is  an  important,  but  often  overlooked, 
step  in  model  building.  Validation  is  necessary  to  determine  how  well  the  selected  models 
predict  using  other  than  fitted  data.  As  such,  one  third  of  the  original  data  collected  was 
set  aside  to  validate  the  predictive  capability  of  the  selected  models.  The  non-fitted  data 
were  entered  into  the  selected  models  and  a  predictive  was  calculated  for  each  model. 
The  predictive  for  each  nxxlel  was  compared  with  the  model's  original  fitted  R^.  The 
smaller  the  difference  between  the  predictive  R^  and  the  fitted  R2,  the  more  robust  the 
model 
.Summarv 

Tius  chapter  described  the  interview  process  that  was  used  to  screen  and 
supplement  the  initial  list  of  factors  presented  in  Chapter  2.  The  analysis  of  variance 
procedures,  which  further  screened  these  factors,  were  also  discussed  as  were  the 
statistical  techniques  employed  in  rtKxlel  development  and  validation.  The  next  chapter 
presents  the  research  results  and  the  analysis  of  those  results  which  answer  the  five 
investigative  questions. 
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IV.  RESULTS 


Overview 

This  chapter  describes  the  results  of  the  research  efforts  detailed  in  Chapter  3.  The 
interview  results  are  presented  first,  followed  by  the  results  of  data  collection,  analysis  of 
variance,  and  model  building. 
faWTVigW  RgSVtIft 

Upon  cranpledon  of  the  twenty-four  interviews,  the  medians  of  the  aggregate 
responses  were  calculated.  If  the  responses  generated  a  median  of  3  (moderate  effect)  or 
greater,  the  factor  was  considered  to  be  significant  and  validated  by  the  interviews.  The 
results  for  each  factor  are  shown  in  Table  2.  Those  factors  deemed  significant  were 
further  screened  to  obtain  a  subset  of  factors  suitable  for  analysis  of  variance.  Factors 
such  as  negotiator  expertise,  workload,  and  priority,  were  deleted  because  data  were  not 
available.  In  some  cases,  this  was  due  to  the  difficulty  in  collecting  data  on  a  factor  and 
other  cases,  the  inability  to  determine  what  type  of  data  should  be  collected  to  accurately 
measure  the  factor.  During  the  interviews,  offsets  and  the  identity  of  the  prime  contractor 
were  omsidaed  significant  by  the  experts  for  disposition  times  but  not  sustention  rates. 
Because  data  were  collected  on  those  factors,  the  statistical  effect  of  offsets  and  the 
identity  of  the  prime  contractor  on  sustention  rates  was  also  assessed. 

Three  of  the  original  factors,  complexity  of  the  issue,  up-firont  time,  and  disposition 
time  were  revised  or  redefined  based  on  interview  discussions.  Complexity  was  renamed 
legal  complexity  to  indicate  the  extent  to  which  the  issue  stretches  the  limits  of  legal 
precedents.  This  name  change  was  based  on  ccxifusion  over  whether  "complexity" 
referred  to  techiucal  or  legal  complexity.  The  consensus  of  the  interviewees  was  that  legal 
complexity  had  a  stronger  effect  on  the  dependent  variables  than  technical  complexity.  As 
such,  data  woe  collected  on  the  legal  complexity  of  the  case,  not  technical  complexity. 
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To  assess  legal  complexity,  legal  opinions,  documented  evaluation  of  litigation  risk,  and 
proposed  contractor  defenses  to  the  proof  elements  were  reviewed. 

The  other  revisions  to  the  original  factors  resulted  in  the  combination  of  up-front 
time  and  disposition  time  into  one  variable  called  disposition  time,  and  the  creation  of  a 
new  variable  called  RPA  revisions.  When  the  survey  was  written,  the  total  time  to  settle  a 
defective  pricing  case  was  broken  into  two  elements:  (1)  up-front  time,  which  was  defined 
as  the  time  between  the  initial  audit  and  the  last  supplement,  and  (2)  disposition  time, 
which  was  defined  in  the  survey  as  the  time  from  the  last  audit  supplement  to  nKxlification 
distribution.  There  was  confusion  over  these  two  factors,  and  the  consensus  of  those 
interviewed  was  that  sustention  rates  are  afiected  by  the  total  time  involved  in  processing 
a  defective  pricing  case.  Therefore,  the  distincticm  between  the  time  elements  was 
eliminated.  Disposition  time  was  redefined  to  encompass  the  entire  time  fit>m  initial  audit 
to  contract  modification.  Up-front  time  was  originally  designed  to  capture  the  time  spent 
issuing  audit  supplements  and  revising  the  RPA.  Because  of  the  confiision  and 
redefinition  of  disposition  time,  up-fix)nt  time  was  changed  to  RPA  revisions,  which 
captures  whether  or  not  the  RPA  was  revised. 

In  addition  to  quantifying  the  effect  of  the  original  factors  listed  on  the  survey,  the 
interviewees  were  asked  to  propose  additional  factors  they  believed  affected  sustention 
rates  (v  disposition  times.  Their  proposed  factors  are  provided  in  Table  2.  Analysis  of  the 
responses  led  to  the  addition  of  one  variable  which  captures  whether  an  interest  charge 
was  assessed.  Otho-  suggested  facrors  were  determined  to  be  partially  encompassed  in 
existing  variables  or  did  not  lend  themselves  to  data  collection. 

Table  2  summarizes  the  interview  results  and  the  subsequent  actions  taken  by  the 
researchers.  The  table  presents  each  of  the  original  variables  and  indicates  whether  it  was 
deemed  significant  or  insignificant  during  the  interviews.  Except  where  otherwise  noted, 
factors  were  significant  or  insignificant  for  both  sustention  rates  and  disposition  times. 
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The  median  fen:  each  factor  is  also  presented  as  are  the  additional  factors  suggested  during 
die  interviews.  A  more  in-depth,  narrative  discussion  of  these  additional  factors  is 
contained  in  Appendix  D.  For  all  factors,  both  original  and  additional,  the  action  taken  by 
the  researchers  (factor  was  deleted,  kept,  redefined,  etc.)  is  described.  Finally,  for  those 
factors  kept  for  statistical  evaluation,  the  variable  names  and  measurement  categories  are 
provided.  A  more  detailed  description  of  these  factors  is  presented  in  Appendix  E. 


TABLE  2 

INTERVIEW  RESULTS 


Survey  Number/ 

Factor 

Interview  Results: 
Medians 

Action 

Variable  Name:  | 

Measurement  Categories  | 

1.1  Alleged  Defect 
Amount 

Significant: 

Rates  S.O/Times  3.0 

Kept 

RPA: 

RPA  <  S200K 

$200K  5  RPA  <  $1M  1 

RPA  >  SIM 

IJZRatioofRPAto 
Audited  Amount 

Insignificant: 

Rates  2.0/Times  2.0 

Deleted 

U  Cost  Element 

Significant: 

Rates  3.0/rimes  3.0 

Kept 

COSTELEM:  Labor,  Material, 
Other,  Multiple 

1.4  Contractor  Level 

Significant: 

Rates  3.0/rimes  4.0 

Kept 

PRIMESUB:  Prime  or 
subcontractor  defective  pricing 

15  Type  of  Contract 
Action 

Insignificant: 

Rales  l.O/Times  1.0 

Deleted 

1.6  CO  Disagreement 
withRPA 

Significant: 

Rates  S.O/Times  4.0 

Kept 

DISAGREE:  PtenegoUation 
Objective  <  RPA  or  not 

1.7  Number  of  Issues 

Significant* 

Rates  3.0/rimes  4.0 

Kept 

ISSUES:  Single  issue  or 
multiple  1 

1.8  Complexity 

Significant 

Rales  4.0/rimes  5.0 

Redefined: 

Lesial  Complexity 

GRAY:  Complex  legal  issues  | 
or  not  1 

1.90fibets 

Significant  for 

Times  Only: 

Rates  2.0/rimes  3.5 

Kept 

OFFSETS:  No  offsets 
proposed,  offsets  proposed  &  1 

accepted  by  CO,  offsets  | 

partially  accepted,  offsets  not 
accepted 

1.10  Recipient  of  Fimds 

Insignificant 

Rates  2.0/Times  2.0 

Deleted 

1.11  Method  of 
Disposilion 

Significant 

Rales  4.0/Times  5.0 

Kept 

METHDISP:  Litigated  or  not 

1.12  Fraud  Investigation 

Significant 

Rates  3.5/rimes  5.0 

Kept 

FRAUD:  Investigated  for  fraud 
or  not 
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TABLE  2  (CONTD) 


1.13  Up-Front  Time 
(defined  in  survey  as  the 
time  between  initial 
audit  &  last  supplement) 

Significant: 

Rates  3.0A'imes  4.0 

Revised:  RPA 
Revisions  (#  of 
audit  revisions 
was  the  key  factor 
of  up-front  time) 

1.14  Disposition  Time 
(defined  in  survey  as  the 
time  between  En^  audit 
suiq)lement  &  modifica¬ 
tion  distribution) 

Insignificant: 

Rates  2.0 

Redefined  as  the 
time  between 
initial  audit  and 
modification  date 

2.1  Contract  Type 

Insignificant: 

Rates  2.0/Times  2.0 

Deleted 

2.2  Contract  Action 

Audit  Amount 

Insignificant: 

Rales  1.0/Times  1.0 

Deleted 

2.3  Contractor  Size 

Insignificant' 

Rates  2.0/rimes  2.0 

Deleted 

2.4  Contract  Cost 
Performance 

Insignificant 

Rates  2.5/rimes  2.0 

Deleted 

2J  Identity  of  Prime 
Contractor 

Significant  for 

Times  Only: 

Rates  2.5/Times  3.0 

Kept 

Significant 

Rates  4.5/Times  4.0 

Deleted:  data  not 
available 

1  3.2  Contract^' 

Significant 

Deleted:  data  not 

■C.  1  1  ,  M  . . . 

Rates  4.0/Times  4.0 

available 

3.3  Organizational 
Structure 

Significant 

Rales  4.0/rimes  5.0 

Redefined: 

Product  Center 

3.4  Workload 
(Centralized  Offices) 

Significant 

Rates  3.0/Times  4.0 

Deleted:  data  not 
available 

3.S  Workload 
(Decentralized  Offices) 

Significant 

Rales  4.0/rimes  5.0 

Deleted:  data  not 
available 

3.6nriority 

(Decentralized  Offices) 

Significant 

Rates  4.0/rimes  5.0 

Deleted:  data  not 
available 

Interest  Assessed 

Factor  suggested  in 
interviews 

Added 

Audit  Characteristics: 
audit  quality/com|dete- 
ness;  auditor  willingness 
to  revise  RPA;  coopeia- 
tion/communication 
between  audits  and  CO 

Factor  suggested  in 
interviews 

Not  included: 
partially  captured 
in  DISAGREE 
andRPA.REVS 

Assist  Audits: 
number  of  subcontractor 
assist  audits  in  support 
of  a  prime  contractor 
audit 

Factor  suggested  in 
interviews 

Not  included: 
partially  captured 
inWUMESUB 
and  ISSUES 

RPA_REVS:  RPA  revised  or 
not 


TIME:  <  1  year,  1  to  2  years,  2 
to  3  years,  >  3  years 


KTR:  Boeing,  General 


Dynamics  (Fort  Worth),  Loral, 
Nfertin  Marietta,  McDonnell 
Douglas,  UT/Piatt  &  Whitney, 
Other 
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TABLE  2  (CONTD) 


Contractor  Attitudes: 
willingness  to  settle, 
agreement  with  RPA, 
netwmking 

Factor  suggested  in 
interviews 

Not  included: 
partially  captured 
inKTR 

Quality  Documentation; 
completeness/availability 
of  government  evidence 

Factor  suggested  in 
interview 

Not  included: 
data  not  available 

Expertise  of 

Participants: 
expertise  of  entire 
defective  pricing  team, 
including  CO,  legal, 
auditms,  technical 
personnel,  etc. 

Factor  suggested  in 
interview 

Not  included: 
data  not  available 

Personnel  Turnover 
tumovOT  of  all  team 
members 

Factm  suggested  in 
interview 

Not  included; 
data  not  available 

Factor  suggest  in 
interview 

Not  included: 
data  not  available, 
but  CENT  may 
provide  insight 

Analysis  of  Variance 

Analysis  of  variance  (ANOVA)  techniques  were  used  to  determine  which  of  the 
factors  for  which  data  were  collected  explain  the  greatest  variation  in  sustention  rates  and 
disposition  times.  The  one-way  analysis  of  variance  results  for  each  factor  tested  are 
presented  in  separate  tables.  Each  table  also  indicates  whether  or  not  there  was  a 
significant  difference  in  the  class  means  at  the  a  =  .05  level.  For  those  factors  with 
significant  differences,  tables  showing  the  means  and  the  results  of  the  Bonferroni 
comparison  of  means  test  for  rates  and  times  are  presented.  For  the  mean  comparisons,  an 
asterisk  {*)  indicates  that  the  means  for  these  classes  are  significantly  different  from  the 
means  for  all  other  classes  at  alpha  a  .05.  Metuis  fOT  classes  which  share  the  same  letter 
do  not  differ  significantly  at  alpha  =  .05. 

Alleged  Defect  Amount  (RPA).  The  results  of  the  one-way  analyses  using  the 
alleged  defect  amount  as  the  predictor  variable  fm*  sustention  rate  and  disposition  time  are 
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presented  in  Table  3.  The  low  p- values  indicate  that,  for  both  sustention  rates  and 
disposition  times,  the  hypothesis  that  the  means  are  equal  for  all  RPA  size  categories  is 
rejected  at  the  a  =  .05  level.  The  R-Squared  values  indicate  that  the  size  of  the  final  RPA 
alone  accounts  for  over  14%  of  the  total  variance  in  sustention  rates  and  10%  of  the 
variation  in  disposition  times. 


TABLE  3 

ALLEGED  DEFECT  AMOUNT-  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Sustention  Rate 

2 

9.56 

.0001 

.142606 

Yes 

Disposition  Time 

2 

6.44 

.0022 

.100718 

Yes 

Note:  Sustoition  rate  model  weighted  by  l/o^  to  equalize  variances. 


As  shown  in  Table  4,  the  Bonferroni  test  revealed  that  the  mean  sustention  rates 
differed  for  each  of  the  size  categories.  For  cases  with  RPAs  of  more  than  $1,000,000, 
the  sustention  rates  were  32%  less  than  those  with  smaller  RPAs.  Similarly,  the  sustention 
rates  for  cases  involving  between  $200,000  and  $1,000,000,  were  less  than  those 
involving  less  than  $200,000.  Note,  however,  tiiat  while  the  difference  in  these  last  means 
was  determined  to  be  statistically  significant,  the  actual  difference  is  only  0.8%.  As  such, 
the  distinction  between  small  and  medium  sized  RPAs  does  not  appear  to  be  as  important 
as  between  small  and  large  sized  RPAs.  In  general,  it  appears  that  cases  with  larger 
allegation  amounts,  especially  over  $1,000,000,  will  experience  lower  sustention  rates. 

For  disposition  times,  the  Bonferroni  test  showed  a  significant  difference  in  the 
mean  of  cases  involving  less  than  $200,000  and  those  involving  more  than  $1,000,000. 
Cases  with  an  RPA  greater  than  $1,000,000  were  found  to  take  an  average  of  557  days 
longer  than  those  with  smaller  RPAs.  No  statistically  significant  difference  was  found 
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between  the  other  categories,  but  the  average  disposition  time  did  increase  as  the  amount 
of  the  RPA  increased.  Therefore,  it  is  concluded  that  the  size  of  the  RPA  is  a  significant 
predictor  of  both  sustention  rates  and  disposition  times. 


TABLE  4 

ALLEGED  DEFECT  AMOUNT  -  MEAN  COMPARISONS 


Class 

#of 

Observations 

Mean  Rates  & 
Comparison  Test 

Mean  Times  & 
Comparison  Test 

<200,000 

58 

89.4* 

762  a 

200,000-1,000000 

38 

88.6* 

958  ab 

^1,000,000 

22 

56.6* 

1397  b 

*  The  means  for  these  classes  are  significantly  different  from  the  means  for  all  other  classes 


at  alpha  =  .05. 

Means  for  classes  with  the  same  letter  do  not  differ  significantly  at  alpha  =  .05. 


Cost  Element  (COSTELEM).  The  results  of  the  one-way  analyses  using  Cost 
Element  as  the  predictor  variable  for  sustention  rate  and  disposition  time  are  presented  in 
Table  5.  The  F-values  and  corresponding  p-values  show  that  there  are  no  significant 
differences  in  the  mean  sustention  rates  or  disposition  times  for  the  various  cost  elements. 
The  very  low  R-squared  further  indicates  that  Cost  Element  does  not  add  to  the 
explanation  of  either  disposition  times  or  sustention  rates.  Therefore,  there  is  no  evidence 
to  support  the  original  hypothesis  that  the  different  cost  elements  introduce  varying  levels 
of  complexity  to  defective  pricing  cases  and  result  in  different  sustention  rates  and 
dispositicMi  times. 
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TABLES 


COST  ELEMENT  -  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Diff.inp? 

Sustention  Rate 

3 

1.8 

.1512 

.045223 

No 

Disposititxi  Time 

3 

.74 

.5289 

.019165 

No 

CfHitractor  Level  (PRIMESUB).  As  shown  in  Table  6,  the  low  p-value  for  the 
sustention  rate  model  allows  rejection  of  the  hypothesis  that  the  mean  sustention  rate  is 
the  same  for  prime  contractors  and  subcontractors.  While  the  F  test  shows  a  significant 
difference  in  the  means,  however,  the  R-Squared  value  of  .073915  indicates  that  this 
difference  accounts  for  only  a  small  portion  of  the  overall  variation  in  sustention  rates. 
For  disposition  times,  the  high  p-value  and  very  low  R-squared  value  indicate  that 
disposition  times  are  not  affected  by  the  level  of  the  contracmr  involved. 


TABLE  6 

CONTRACTOR  LEVEL  -  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Diffinn? 

Sustention  Rate 

1 

9.26 

.0029 

.073915 

Yes 

Disposition  Time 

1 

.83 

.3652 

.007074 

No  1 

Note;  Sustentkm  rate  model  weighted  by  l/o^  to  equalize  variances. 


Before  conducting  this  research,  the  contractor  level  was  believed  to  impact 
sustention  rates,  but  the  direction  of  that  impact  was  unknown.  Table  7  shows  that  the 
sustention  rates  for  cases  involving  defective  pricing  at  the  prime  contractor  level  are  an 
average  of  23.8%  higher  than  those  involving  defective  pricing  at  the  subcontractor  level. 
For  disposition  times,  the  hypothesis  was  that  subcontractor  defective  pricing  cases  would 
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take  longer  to  resolve  than  prime  contractor  cases.  This  hypothesis  was  not  supported  by 
the  data  which  reflected  no  significant  differences  in  disposition  times. 


TABLE? 

CONTRACTOR  LEVEL  -  MEAN  COMPARISONS 


#of 

Mean  Rates  & 

Mean  Times  &  | 

Class 

Observations 

Comparism  Test 

Comparistm  Test  I 

Prime 

93 

■HIDH 

N/A 

Subcontractor 

25 

I 

*  The  means  to  these  classes  are  significantly  diftoent  from  the  means  to  all  other  classes 
at  alpha  =  .05. 


CO  Disagreement  with  RPA  (DISAGREE).  Table  8  shows  the  results  of  the 
one-way  analyses  using  DISAGREE  as  the  predictor  variable  for  sustention  rate  and 
disposition  time.  For  sustention  rates,  the  p-value  of  .0001  indicates  a  significant 
difference  in  the  means.  In  addition,  the  very  high  R-squaied  value,  0.3405,  means  that 
this  variable  is  one  of  the  most  significant  predictors  of  sustention  rates.  It  is  therefore 
concluded  that  contracting  officer  agreement  with  the  audit  report,  measured  by  whether 
or  not  the  negotiation  objective  matches  the  final  RPA,  has  a  very  significant  effect  on 
sustention  rates. 

This  DISAGREE  variable  does  not  seem  as  important,  however,  for  disposition 
times.  The  high  p-value  indicates  that  the  means  for  disposition  time  do  not  differ  based 
on  agreement  with  the  RPA.  The  low  R-squared  value  confirms  that  this  variable  adds 
little  to  the  prediction  or  explanation  of  disposition  times.  This  indicates  that  while  the 
contracting  officer  may  recover  fewer  dollars  as  a  result  of  the  disagreement  with  the  audit 
report,  the  case  does  not  take  longer  to  settle.  Surprisingly,  the  expectation  that  such 
disagreement  would  lengthen  the  disposition  times  because  of  the  additional  time  spent 
attempting  to  reconcile  the  government's  position  was  not  confirmed. 
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TABLES 


CO  DISAGREEMENT  WITH  RPA  -  ANOV A  RESULTS 


Model 

DF 

F-Value 

P-Value 

Sustention  Rate 

1 

80.76 

.0001 

.340464 

Yes 

Disposition  Hme 

1 

2.65 

.1063 

.016655 

No 

As  shown  in  Table  9,  the  expectation  that  CO  disagreement  with  the  RPA  will  lead 
to  lower  sustention  rates  is  confiimed  by  the  data.  On  average,  cases  involving 
disagreement  lead  to  a  36.8%  lower  sustention  rate. 


TABLE  9 

CO  DISAGREEMENT  WITH  RPA  -  MEAN  COMPARISONS 


1 

#of 

Mean  Rates  & 

Mean  Times  & 

1  Class 

Observations 

Ccxnparison  Test 

Comparison  Test 

Yes 

29 

N/A 

1  No 

89 

*  The  means  for  these  classes  are  significantly  different  from  the  means  fw  all  other  classes 


at  alpha  =  .05. 

Number  of  Issues  (ISSUES).  Table  10  displays  the  ANOVA  results  for  Number  of 
Issues.  For  sustention  rates,  the  hypothesis  that  the  mean  sustention  rate  is  equal  whether 
there  were  single  or  multiple  issues  is  rqected  at  a  =  .05  level.  While  the  difference  in 
means  was  found  to  be  significant,  the  low  R-Squared  value  indicates  that  the  number  of 
issues  explains  only  a  small  portion  (6%)  of  the  total  variation  in  sustention  rates.  As 
such,  the  number  of  issues  involved  may  not  be  a  significant  predictor  variable.  The 
smtistics  for  disposition  times  indicate  a  significant  difference  in  the  mean  disposition  times 
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between  cases  with  single  or  multiple  issues  and  that  the  variable  ISSUES  does  add  to  the 
explanation  and  prediction  of  disposition  times. 


TABLE  10 

NUMBER  OF  ISSUES  -  ANOVA  RESULTS 


Model 

DF 

F- Value 

P- Value 

Diff.  inji?  1 

Sustention  Rate 

1 

7.71 

.0064 

.062287 

Yes 

Disposition  Time 

1 

15.77 

.0001 

.119704 

Yes 

The  mean  comparisons  in  Table  1 1  confirm  the  behef  that  cases  involving  multiple 
issues  have  lower  sustention  rates  and  take  longer  to  settle  than  those  with  single  issues. 


TABLE  11 

NUMBER  OF  ISSUES  -  MEAN  COMPARISONS 


Class 

#of 

Observations 

Mean  Rates  & 
Comparison  Test 

Mean  Times  & 
Comparison  Test 

Stride 

77 

758  * 

Multiple 

41 

73.8  * 

1293* 

*  The  means  for  these  classes  are  significantly  diffoent  from  the  means  for  all  other  classes 


at  alpha  =  .05. 

Legal  Complexity  (GRAY).  This  variable  categorized  cases  as  legally  complex  or 
not  complex.  Determining  factors  included  the  presence  of  opinions  commenting  on  the 
legal  sufficiency  of  contractor  and  auditor  rebuttals,  documented  evaluation  of  litigation 
risk,  strong  contractor  defenses  to  the  required  proof  elements,  and  an  overall  assessment 
of  the  legal  difficulties  in  meeting  the  burden  of  proof  in  each  case.  The  results  of  the  one¬ 
way  analyses  using  GRAY  as  the  predictor  variable  for  sustention  rate  and  disposition 
time  are  presented  in  Table  12.  The  p- values  for  both  models  are  .0001,  indicating  great 
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c(mfulence  that  mean  sustention  rates  and  disposition  times  vary  based  on  the  legal 
complexity  of  the  issues  involved.  The  high  R-squared  values  indicate  that  the  legal 
complexity  of  the  case  accounts  for  over  28%  of  the  variation  in  sustention  rates  and 
almost  20%  of  the  variation  in  disposition  times. 


TABLE  12 

LEGAL  COMPLEXITY  -  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Diff.  in  |i? 

Sustention  Rate 

1 

45.86 

.0001 

.283314 

Yes 

Disposititm  Time 

1 

28.58 

.0001 

.197653 

Yes 

Vole:  Both  models  weighted  by  l/o^  to  equalize  variances. 


Table  13  shows  that  the  average  sustention  rate  is  much  lower  and  the  average 
disposition  time  is  significantly  longer  for  defective  pricing  cases  involving  gray  issues.  In 
both  cases,  the  data  contirm  the  original  hypotheses. 

TABLE  13 

LEGAL  COMPLEXITY  -  MEAN  COMPARISONS 


#of 

Mean  Rates  & 

Mean  Times  & 

Class 

Observations 

Comparison  Test 

Compaiisoi  Test 

Complex 

SSI 

1398  * 

Not  Complex 

664  * 

*  The  means  fw  these  classes  are  signiflcantly  diffnent  from  the  means  for  all  other  classes 
at  alpha  =  .05. 


Offsets  (OFFSETS).  The  ANOVA  results  in  Table  14  indicate  that  the  hypothesis 
of  equal  means  can  be  rejected  for  sustention  rates,  but  not  fen:  disposition  times.  These 
results  are  consistent  with  the  R-squared  values  which  are  high  for  sustention  rate  and  low 
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for  disposition  time.  As  such,  the  presence  of  offsets  is  helpful  in  explaining  and 
predicting  sustention  rates,  but  not  disposition  times. 


TABLE  14 

OFFSETS-ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Diff.  inp.?  j 

Sustention  Rate 

3 

12.48 

.0001 

.247300 

Yes 

Disposition  lime 

3 

1.75 

.1617 

.043920 

No 

Note:  Sustoition  rate  mode 

weighted  by  l/o^  to  equal 

lize  variances. 

The  BtMiferroni  results  presented  in  Table  15  reveal  significant  differences  in  the 
sustention  rates  for  all  offset  categories.  This  means  that  not  only  is  there  a  difference 
triggered  by  the  proposal  of  an  offset,  there  are  also  differences  depending  on  the 
contracting  officer's  evaluation  of  that  offset 

Cases  in  which  offsets  are  not  accepted  have  the  highest  sustention  rates.  Fully 
accepted  offsets  have  the  next  highest  sustention  rates,  followed  by  cases  in  which  no 
offsets  were  proposed.  The  lowest  sustention  rates  were  cases  in  which  the  offsets  were 
(Mily  partially  accepted  by  the  contracting  officer. 


TABLE  15 

OFFSETS  -  MEAN  COMPARISONS 


Gass 

#of 

Observations 

Mean  limes  & 
Comparison  Test 

Mean  Times  & 
Comparison  Test 

Not  Proposed 

81.4  • 

N/A 

Accepted  in  Full 

93.9  • 

Accepted  in  Part 

18 

73.7  * 

7 

98.9  • 

*  The  means  fm  these  classes  are  significantly  different  from  the  means  for  ail  other  classes 
at  alpha  =  .05. 
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Method  of  Disposition  (METHDISP).  Table  16  presents  the  ANOVA  results 
using  METHDISP  as  the  predictor  variable  for  sustention  rate  and  disposition  time.  For 
this  variable,  the  F-tests  show  that  at  the  a  =  .05  level,  the  average  sustention  rate  and 
disposition  time  differ  depending  on  whether  or  not  the  case  was  litigated.  The  R-squared 
value  of  .126631  confirms  the  importance  of  the  method  of  disposition  in  explaining 
sustention  rates.  The  R-squared  value  for  disposition  time  is  surprisingly  low,  however, 
given  die  large  difference  in  average  time  shown  in  Table  17.  This  anomaly  is  caused  by 
the  fact  that  only  three  of  the  cases  analyzed  were  litigated;  it  is  difficult  to  perform 
meaningful  statistical  analysis  with  so  few  data  points. 


TABLE  16 

METHOD  OF  DISPOSITION  -  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Sustention  Rate 

1 

16.82 

.0001 

.126631 

Yes  1 

Disposition  Time 

1 

9.63 

.0024 

.076654 

Yes  1 

As  shown  in  Table  17,  litigated  cases  take  an  average  of  1295  days  longer  to  resolve 
than  cases  setded  without  litigation.  In  addition,  sustention  rates  are,  on  average,  almost 
75%  lower  for  litigated  cases. 


TABLE  17 

METHOD  OF  DISPOSITION  -  MEAN  COMPARISONS 


#of 

Mean  Rates  & 

Mean  Times  & 

Class 

Observations 

Comparison  Test 

Comparison  Test 

Litigated 

2206  * 

1 

911  * 

*  The  means  fOT  these  classes  are  significantly  different  from  the  means  for  all  other  classes 
at  alpha  » .05. 
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Fraud  Investigation  (FRAUD).  As  shown  in  Table  18,  the  F-values  and 
corresponding  p-values  indicate  that  fraud  investigations  do  not  lead  to  statistically 
different  sustention  rates  or  disposition  times.  The  very  low  R-squared  values  confirm  the 
fact  that  FRAUD  does  not  add  to  the  prediction  or  explanation  of  either  disposition  times 
or  sustention  rates. 


TABLE  18 

FRAUD  INVESTIGATION  -  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

msi 

Sustention  Rate 

1 

.04 

.8382 

.000361 

No 

Dispositicxi  Hme 

1 

2.32 

.1305 

.019600 

No 

The  analysis  of  variance  conclusions  are  scHnewhat  surprising  given  the  visibly  large 
difference  in  the  mean  disposition  times.  The  results  show  that  cases  investigated  for 
fraud  took  an  average  of  436  days  longer  to  resolve  than  those  not  investigated.  Possible 
explanations  for  this  iactor  being  visibly,  but  not  statistically  significant,  include  the 
reduced  power  of  the  F-test  caused  by  the  low  number  of  data  points  involving  fraud 
investigation  (7)  and  the  high  standard  deviatirai  of  that  data  (670  days). 

Disposition  Time  (TIME).  The  ANOVA  results  using  Disposition  Time  as  the 
predictor  variable  for  sustention  rate  are  presented  in  Table  19.  At  the  a  =  .05  level,  the  F 
test  indicates  that  there  are  differences  in  the  means.  The  R-squared  value  indicates  that 
not  only  are  there  differences  in  the  means,  this  variable  explains  over  10%  of  the  overall 
variation  in  sustention  rates. 
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TABLE  19 


DISPOSITION  TIME  -  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Diff.  in  p. 

Sustention  Rate 

3 

4.45 

.0054 

.104900 

Yes 

Disposititm  Hme 

N/A 

N/A 

N/A 

N/A 

N/A 

Note:  Sustention  rate  model 

weighted  by  l/o^  to  equal 

lize  variances. 

The  Bonfenoni  test  results  shown  in  Table  20  indicate  significant  differences 
between  all  of  the  time  categories,  with  sustendtHi  rate  decreasing  as  disposition  time 
increases.  These  results  confirm  the  expectation  that  longer  disposition  times  are 
associated  with  lower  sustention  rates. 


TABLE  20 

DISPOSITION  TIME  -  MEAN  COMPARISONS 


Class 

#of 

Observations 

Mean  Rates  & 
Comparison  Test 

Mean  Times  & 
Comparison  Test 

<  1  Year 

32 

N/A 

1-2  Years 

23 

2-3  Years 

24 

78.9  * 

>3  Years 

39 

74.8  • 

*  The  means  for  these  classes  are  significantly  different  from  the  means  for  all  other  classes 
at  alpha  =  .05. 


Identity  of  the  Prime  Contractor  (KTR).  Table  21  displays  the  one-way 
ANOVA  results  for  the  variable  KTR.  The  p-values  of  0.0001  for  both  models  indicate 
that  the  mean  sustention  rate  and  disposition  time  for  at  least  one  contractor  is  statistically 
different  than  the  others.  The  high  R-squared  values  confirm  that  Identity  of  the  Prime 
Contractor  is  a  critical  variable  in  the  prediction  and  explanation  of  sustention  rates. 
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TABLE  21 


IDENTITY  OF  PRIME  CONTRACTOR  -  ANOVA  RESULTS 


Model 

DF 

F- Value 

P-Value 

Diff.inn?  1 

Sustention  Rate 

6 

8.62 

.0001 

.317939 

Yes  1 

Dispositimi  Time 

6 

5.07 

.0001 

.215014 

Yes  1 

Note:  Sustention  rate  model  weighted  by  l/o^  to  equa 

lize  variances. 

To  further  investigate  the  differences  between  contractors,  the  Bonferroni 
comparison  of  means  test  results  are  displayed  in  Table  22.  For  sustention  rates,  almost 
all  of  the  differences  among  contractors  are  significant.  The  exceptions  to  this  are  the 
difference  between  Loral  and  Martin  Marietta  and  the  difference  between  United 
Technologies  and  Other.  Using  the  data  analyzed,  the  contractors  can  be  listed  as  follows 
in  ascending  order  of  sustention  rates:  (1)  Boeing;  (2)  United  Technologies/Pratt  & 
Whimey;  (3)  Other  (4)  Martin  Marietta;  (5)  Loral;  (6)  McDonnell-Douglas;  and  (7) 
General  Dynamics. 

For  disposition  times,  the  Bonferroni  test  indicated  that  significant  differences  exist 
for  all  contractors.  Using  the  data  analyzed,  the  contractors  can  be  listed  as  follows  in 
ascending  order  of  average  disposition  time:  (1)  McDonnell-Douglas;  (2)  United 
Technologies/Pratt  &  Whimey;  (3)  Boeing;  (4)  General  Dynamics;  (5)  LOTal;  and  (6) 
Other  (7)  Martin  Marietta. 
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TABLE  22 

IDENTITY  OF  PRIME  CONTRACTOR  -  MEAN  COMPARISONS 


Class 

#of 

Observations 

Mean  Rates  & 
ComparisOT  Test 

Mean  Dmes  & 
Comparisoi  Test 

Boeing 

11 

59.6  a 

768  a 

Ckneral  Dynamics 

21 

102.1  b 

860  b 

Loral 

10 

85.2  c 

919  c 

Martin  Marietta 

10 

84.9  c 

1608  d 

McDonnell-Douglas 

13 

88.0  d 

418  e 

United  Tech/P  &W 

10 

77.7  e 

761  f 

Other 

43 

78.6  e 

1082  g 

Note:  Kfeans  to  classes  with  the  same  Iett»  do  not  differ  significantly  at  alpha  =  .05. 


Product  Center  (CENTER).  As  shown  in  Table  23,  the  F-test  for  the  sustention 
rate  model  has  a  p-value  of  0.0121.  Therefore,  at  the  a  =  .05  level,  the  hypothesis  that  the 
mean  sustention  rate  is  equal  for  all  three  centos  is  rejected.  For  disposition  times,  the 
hypothesis  of  equal  means  is  also  rejected.  Die  R-squared  values  indicate  that  Product 
Ctoter  explains  7.4%  of  the  variation  in  sustention  rates  and  12%  of  the  variation  in 
disposition  times.  Therefore,  this  variable  may  be  a  better  predictor  for  disposition  times 
than  sustention  rates 


TABLE  23 

PRODUCT  CENTER  -  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Sustention  Rate 

2 

4.59 

.0121 

.073936 

Yes 

Disposition  Dme 

2 

7.84 

.0006 

.119934 

Yes 

As  shown  in  Table  24,  the  Bonferroni  comparison  of  means  test  revealed  that  the 
mean  sustentirai  rates  for  ESC  were  significantly  different  than  those  for  ASC.  No 
significant  differences  were  revealed  when  comparing  SMC  with  ESC  and  ASC.  As  with 
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sustenticHi  rates,  the  Bonfenoni  test  showed  a  significant  difference  in  the  mean 
disposition  times  of  ASC  and  ESC,  with  ESC  cases  taking  longer.  No  significant 
differences  were  found  when  comparing  the  disposition  times  for  SMC  and  ESC  or  SMC 
and  ASC.  These  results  lead  to  the  conclusion  that  the  product  center  resolving  the 
defective  pricing  case  does  affect  both  rates  and  times.  On  average,  ESC  cases  result  in 
lower  sustention  rates  and  longer  disposition  times  than  ASC  cases. 

TABLE  24 

PRODUCT  CENTER  -  MEAN  COMPARISONS 


Class 

#of 

Observations 

Mean  Rates  & 
Comparison  Test 

Mean  Times  & 
Comparison  Test 

ASC 

85 

86.7  a 

857  a 

ESC 

14 

63.9  b 

1638  b 

SMC 

19 

80.7  ab 

820  ab 

Note:  Means  fw  classes  with  the  same  letter  do  not  differ  significantly  at  alpha  =  .05. 


Interest  Assessed  QINTEREST).  The  ANOVA  results  for  the  variable  INTEREST 
are  presented  in  Table  25.  The  p-values  for  both  models  suggest  that  at  the  a  =  .05  level, 
the  hypothesis  that  the  mean  sustention  rate  and  mean  disposition  time  are  equal 
regardless  of  whether  or  not  an  interest  charge  is  assessed  is  rejected.  While  the  R- 
squared  values  are  relatively  low,  this  variable  does  contribute  somewhat  to  the 
explanation  of  both  sustention  rates  and  disposition  times. 


TABLE  25 

INTEREST  ASSESSED  -  ANOVA  RESULTS 


Model 

DF 

F-Value 

P-Value 

Diff.  inti?  1 

Sustention  Rate 

1 

9.03 

.0033 

.072203 

Yes  1 

Disposititxi  Tune 

1 

4.88 

.0291 

.040387 

Yes  1 

Note:  Disposition  Time  model  weighted  by  l/o^  to  equalize  variances. 
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The  mean  comparisons  in  Table  26  show  that  sustention  rates  are  lower  and 
disposition  times  shorter  for  cases  in  which  interest  was  assessed.  The  disposition  time 
data  provide  evidence  that  the  TINA  amendment  assessing  interest  appears  to  have  been 
effective  in  motivating  faster  settlement  of  defective  pricing  cases. 


TABLE  26 

INTEREST  ASSESSED  -  MEAN  COMPARISONS 


#of 

Mean  Rates  & 

Mean  Times  & 

Cass 

Observations 

Ccxnpaiison  Test 

Comparison  Test 

Yes 

56 

90.7  * 

No 

62 

76.1  * 

*  The  means  f<x'  these  classes  are  significantly  different  from  the  means  for  all  other 


classes  at  alpha  =  .05. 

RPA  Revisioiis  (RPA_REVS).  The  results  of  the  one-way  analyses  using 
RPA.REVS  as  the  predictor  variable  for  sustention  rate  and  disposition  time  are 
presented  in  Table  27.  For  this  variable,  there  is  a  significant  difference  in  the  means  for 
disposition  times  but  not  for  sustention  rates.  Similarly,  the  R-squared  values  indicate  that 
whether  ot  not  the  RPA  has  been  revised  explains  about  1 1%  of  the  variation  in 
disposition  times  and  less  than  1%  of  the  variation  in  sustention  rates. 


TABLE  27 

RPA  REVISIONS  -  ANOVA  RESULTS 


Model 

DF 

F- Value 

P-Value 

Diff.  in  |x? 

Sustention  Rate 

1 

1.03 

3\\1 

.008820 

No 

Disposition  Tune 

1 

14.72 

.0002 

.112592 

Yes  1 

56 


In  TaUe  28,  the  comparison  of  means  data  confirm  the  expectation  that  disposition 
tiroes  are  longer  when  the  RPA  is  revised.  While  these  revisions  may  be  necessary,  the 
evidmce  shows  that  they  do  lengthen  the  disposition  time  by  an  average  of  almost  500 
days. 


TABLE  28 

RPA  REVISIONS  -  MEAN  COMPARISONS 


Mean  Rates  & 

Mean  Times  &  I 

Class 

0  'ervauOTS 

Comparison  Test 

Comparison  Test  | 

Yes 

j7 

N/A 

1199* 

No 

61 

705  *  1 

*  The  means  for  these  classes  are  significantly  different  from  the  means  for  all  other 
classes  at  alpha  =  .OS. 


Summary  of  Analvsi.s  of  Variance  Results.  The  analysis  of  variance  results  for  each 
of  the  factors  considered  to  affect  sustention  rates  and  disposition  times  were  presented 
above.  The  results  of  the  F*tests  and  analysis  of  the  R-squared  values  were  used  to  assess 
the  extent  to  which  each  variable  contributes  to  the  explanation  of  sustention  rates  and 
disposition  times.  The  Bonfenoni  comparison  of  means  tests  for  sustention  rates  and 
disposition  times  for  all  factors  are  summarized  in  Tables  29  and  30. 


57 


TABLE  29 


SUSTENTION  RATE  MEAN  COMPARISONS 


Factor 

Class 

#ofObs 

Mean  Comparison  Test 

Alleged  Defect  Amount 

<200,000 

58 

89.4* 

200,000-1,000000 

38 

88.6* 

^1,000,000 

22 

56.6* 

Cost  Element 

Material 

61 

87.3  a 

Labmr 

13 

85.8  a 

Other 

13 

73.7  a 

31 

82.5  a 

Contractor  Level 

Prime 

93 

88.1  * 

Subcontractor 

25 

64.3  * 

CO  Disagreemoit  with  RPA 

Yes 

29 

55.3  * 

No 

89 

92.1  * 

Numbo’ of  Issues 

77 

88.0  * 

41 

73.8  * 

Legal  Complexity 

Complex 

laHi 

62.9  * 

Not  Comolex 

95.5  * 

OflEsets 

Not  proposed 

76 

81.4  * 

Accepted  in  Full 

17 

93.9  * 

Accqited  in  Part 

18 

73.7  * 

Not  Accepted 

7 

98.9  * 

Method  of  Disposition 

Litigated 

23.3  * 

Not  Litigated 

84.6  * 

Fraud  Investigation 

Yes 

7 

81  a  J 

No 

111 

83.2  a 

RPA  Revisions 

Yes 

57 

85.7  a 

No 

61 

80.6  a 

Disposition  Time 

<  1  Year 

94.3  * 

1-2  Years 

85.8  * 

2-3  Years 

24 

78.9  * 

>3  Years 

39 

74.8  * 

Identity  of  Prime  Contractor 

Boeing 

11 

59.6  a 

General  Dynamics 

21 

102.1  b 

Loral 

10 

85.2  c 

Martin  Marietta 

10 

84.9  c 

McDonnell-Douglas 

13 

88.0  d 

United  Tech/P  &W 

10 

77.7  e 

Other 

43 

78.6  e 

Product  Center 

ASC 

85 

86.7  a 

ESC 

14 

63.9  b 

. 

SMC 

19 

80.7  ab 

Interest  Assessed 

Yes 

56 

90.7  • 

No 

62 

76.1  * 

*  The  means  for  these  classes  are  significantly  different  from  the  means  for  all  other  classes  at 


alpha  =  05. 

Means  for  classes  with  the  same  letter  do  not  differ  significantly  at  alpha  =  .05. 
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TABLE  30 


DISPOSITION  TIME  MEAN  COMPARISONS 


Factor 

Class 

#ofObs 

Mean  Comparison  Test  I 

Alleged  Defect  Amount 

<200,000 

58 

762  a 

200,000-1,000000 

38 

958  ab 

>1,000,000 

22 

1397  b 

Cost  Element 

Material 

61 

855  a 

Labor 

13 

1025  a 

Other 

13 

930  a 

31 

1089  a 

Contractor  Level 

Prime 

93 

911  a 

Subcontractor 

25 

1063  a 

CO  Disagreement  with  RPA 

Yes 

29 

1110  a 

No 

89 

889  a 

Number  of  Issues 

77 

41 

Legal  Complexity 

Complex 

1398  * 

Not  Complex 

664  * 

OfEs^ 

Not  proposed 

76 

846  a 

Accept^  in  Full 

17 

958  a 

Accepted  in  Part 

18 

1199  a 

Not  Accepted 

7 

1305  a 

Method  of  Disposition 

Litigated 

3 

Not  Litigated 

115 

Fraud  Investigation 

Yes 

7 

1354  a 

No 

111 

918  a 

RPA  Revisions 

Yes 

57 

1199  * 

No 

61 

705  • 

Identity  of  Prime  Contractor 

Boeing 

RHI 

768  a 

General  Dynamics 

860  b 

Loral 

919  c 

Martin  Marietta 

1608  d 

McDonnell-Dougias 

418  e 

United  Tech/P  &  W 

761  f 

Other 

1082  g 

Product  Center 

ASC 

85 

857  a 

ESC 

14 

1638  b 

SMC 

19 

820  ab 

Intnest  Assessed 

Yes 

56 

791  * 

No 

62 

1081* 

*  The  means  fcH*  these  classes  are  significantly  difFerait  iiom  the  means  for  all  other  classes  at 


alpha  =  .05. 

Means  for  classes  with  the  same  letter  do  not  differ  significantly  at  alpha  =  .05. 
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Having  identified  a  set  of  factors  for  sustention  rates  and  disposition  times  through 
the  interview  process,  the  factors  were  further  screened  through  an  analysis  of  variance. 
The  factors  having  means  significantly  different  from  one  another  at  the  95%  confidence 


level  became  the  initial  set  of  independent  variables  considered  for  model  inclusion. 
Disposition  Time  was  subsequendy  eliminated  as  a  potential  independent  variable  for  the 
sustention  rate  model.  Because  the  value  of  Disposition  Time  is  unknown  until  the  case 
has  been  settled,  it  is  not  useful  for  prediction.  Table  31  summarizes  the  lists  of  factors 
which  were  used  in  regression  analysis  for  sustention  rates  and  disposition  tin^s. 


TABLE  31 

FACTORS  CONSIDERED  IN  MLR  MODELS 


Statistically  Significant  Factras 
for  Sustention  Rates 

Statistically  Significant  Factors 
for  Disposition  Times 

Alleged  Defect  Amount 

Alleged  Defea  Amount 

Number  of  Issues 

Number  of  Issues 

PT^T  ITI 11’—— 

Method  of  Disposition 

Method  of  Disposition  | 

Identity  of  Prime  Contractor 

Identity  of  Prime  Contractor 

Product  Center 

Product  Center 

Interest  Assessed 

Interest  Assessed 

Contractor  Level 

RPA  Revisions 

Offsets 

CO  Disagreement  with  RPA 

1 
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Best  Subset  Selection.  The  predictor  variables  selected  using  Miller's  method,  as 
well  as  preliminary  regression  results,  are  shown  in  Tables  32  and  33  for  sustention  rates 
and  disposition  times,  respectively.  From  research  conducted  on  Miller's  Method,  it  was 
expected  that  a  set  of  variables  would  consistendy  be  chosen  as  the  best,  but  this  did  not 
happen.  Because  there  were  no  clear  winners  from  the  initial  screening,  a  second 
screening  was  performed.  This  additional  screening  considered  only  those  variables 
selected  by  the  initial  screening.  From  this  additional  screening,  the  models  were  selected 
based  on  the  set  of  variables  which  achieved  the  highest  selection  rate.  A  summary  of  the 
results  for  both  screenings  is  included  in  Appendix  E. 

TABLE  32 

LINEAR  REGRESSION  RESULTS  FOR  SUSTENTION  RATES 


PREDICTOR 


VARIABLES 

COEFFICIENT 

STD  ERROR 

STUDENTS  T 

P 

VIF 

CONSTANT 

99.1594 

2.06389 

48.04 

0.0000 

DISAGREE 

-25.9559 

4.21653 

-6.16 

0.0000 

1.3 

GRAY 

-12.7182 

3.80535 

-3.34 

0.0013 

1.4 

METHDISP 

-42.3987 

10.4860 

-4.04 

0.0001 

1.1 

RPA_LRG 

-20.0849 

5.05466 

-3.97 

0.0002 

1.3 

CENT_ESC 

-23.6567 

6.07031 

-3.90 

0.0002 

1.2 

KTR_MA 

21.4356 

5.99479 

3.58 

0.0006 

1.2 

R-SQUARED 

0.7339 

RESID.  MEAN  SQUARE  (MSE) 

ADJUSTED  R-SQUARED 

0.7117 

STANDARD  DEVIATION 

SOURCE 

DF 

SS 

MS 

F 

P 

REGRESSION 

6 

38265.0 

6377.50  33.09 

0.0000 

RESIDUAL 

72 

13876.3 

192.727 

TOTAL 

78 

52141.4 

192.727 

13.8826 


TABLE  33 


UNEAR  REGRESSION  RESULTS  FOR  DISPOSITION  TIMES 


raEDICTOR 

VARIABLES 

COEFFICIENT 

STD  ERROR 

STUDENT'S  T 

P 

VIF 

CONSTANT 

481.886 

103.139 

4.67 

0.0000 

GRAY 

441.139 

149.013 

2.96 

0.0041 

1.3 

RPA_REVS 

445.602 

136.867 

3.26 

0.0017 

1.2 

CENT_ESC 

636.470 

232.727 

2.73 

0.0078 

1.1 

KTR_MC 

-577.303 

207.853 

-2.78 

0.0070 

1.1 

RPA_LRG 

555.329 

208.065 

2.67 

0.0094 

1.3 

R-SQUARED  0.4683  RESID.  MEAN  SQUARE  (MSE) 

ADJUSTED  R-SQUARED  0.4319  STANDARD  DEVIATION 


3.200E+05 

565.680 


SOURCE 

DF 

SS 

MS 

F 

P 

REGRESSION 

5 

2.057E+07 

4.115E+06 

12.86 

0.0000 

RESIDUAL 

73 

2,336E+07 

3.200E+05 

TOTAL 

78 

4.393E+07 

Verificarion  of  Model  Assuiqptions.  The  tests  to  check  for  assumption  violations, 
as  outlined  in  Qiapter  3,  include  tests  for  heteioscedasticity,  ntm-normality  of  residuals, 
and  multicollinearity.  The  following  results  were  found: 

Heteroscedastiritv.  To  determine  if  the  residuals  had  constant  variances,  the 
residuals  were  plotted  against  the  predicted  values.  The  residual  plot  for  sustention  rates, 
shown  in  Figure  1,  has  no  discernible  trend.  Therefore,  the  assumption  of  constant 
variances  is  considered  valid.  The  plot  for  disposition  times,  shown  in  Figure  2,  however, 
appears  to  have  a  strong  expanding  trend.  This  pattern  occurs  when  the  error  component 
increases  as  the  mean  increases. 
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Hgure  1.  Residual  Plot  for  Sustention  Rates 
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Figure  2.  Residual  Plot  for  Disposition  Times 


To  stabilize  the  residual  variance  for  disposition  times,  a  transformation  of  the 
dependent  variable  (Y)  is  required.  Based  on  the  trend  displayed,  ^  and  log  (Y)  are  the 
two  most  suited  transformations  (McQave  &  Bensen  685).  Transformation  Vt  did  not 
improve  the  residual  plots,  but  log  (Y)  appeared  to  stabilize  the  residual  variation.  The 
logarithmic  transformadon  improved  the  residual  plots  and  R^,  but  invalidated  the 
selection  of  the  CENT_ESC  variable.  Because  the  p-value  of  CENT_ESC  increased  from 
0.0078  to  0.2048  after  the  transformation,  CENT_ESC  was  no  longer  a  useful  predictor 
variable.  To  explain  why  this  might  have  occurred,  a  regression  line  of  the  predictor 
variable  CENT.ESC  was  plotted  before  and  after  the  logarithmic  transformation.  The 
regression  lines  showed  that  CENT_ESC  was  initially  linear,  but  the  logarithmic 
transformation  caused  CENT_ESC  to  become  a  nonlinear  predictor.  Therefore, 
CENT_ESC  was  eliminated  from  the  disposition  time  model.  Table  34  displays  the  linear 
regression  results,  without  the  predictor  variable  CENT_ESC,  of  the  logarithmic  model 
ccHTecting  for  heteroscedasticity.  Figure  3  shows  the  corrected  residual  plot  for  the 
regression  model  displayed  in  Table  34. 
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TABLE  34 


LINEAR  REGRESSION  RESULTS  FOR  DISPOSITION  TIMES 
WITH  LOGARITHMIC  TRANSFORMATION 


PREDICTOR 

VARIABLES 

COEFnCIENT  STD  ERROR 

STUDENTS  T 

P 

VIF 

CONSTANT 

2^091 

0.05882 

43.19 

0.0000 

caiAY 

0.25417 

0.08455 

3.01 

0.0036 

1.2 

RPA.REVS 

0.33446 

0.07655 

4.37 

0.0000 

1.1 

KTR_MC 

-0.47130 

0.11821 

-3.99 

0.0002 

1.1 

RPA.LRG 

0.37985 

0.11874 

3.20 

0.0020 

1.3 

R-SQUARED 

0.4849 

RESID.  MEAN  SQUARE  (MSE) 

0.10422 

ADJUSTED  R-SQUARED 

0.4570 

STANDARD  DEVIATION 

0.32284 

SOURCE 

DF 

SS 

MS 

F 

P 

REGRESSION 

4 

7.25966 

1.81491  17.41 

0.0000 

RESIDUAL 

74 

7.71284 

0.10422 

TOTAL 

78 

14.9725 

HeigreaBioa  Residiial  Pbt 
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Hgure  3.  Residual  Plot  for  Transformed  Disposition  Times 
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Normal  Distribution  of  Residuals.  The  test  for  non-normality  of  residuals  was 
acc(»nplished  by  constructing  a  Wilk-Shapiro/Rankit  Plot.  The  rankit  plot  for  disposition 
times  is  shown  in  Figure  4.  Because  the  plots  appear  to  be  in  a  relatively  straight  line  and 
W  Statistic  0.9524  is  greater  than  the  selected  standard  of  0.9000,  the  assumption  of 
ncnrmally  distributed  residuals  is  not  rejected. 


Wilk-Sbapiro  /  Rankit  Plot 


Figure  4.  Wilk-Shapiro/Rankit  Plot  for  Transformed  Disposition  Times 


The  rankit  plot  for  sustention  rates  is  shown  in  Figure  5.  Although  W  Statistic  0.9072  > 
0.9000,  the  plots  do  not  conform  well  to  a  straight  line.  To  better  understand  why  the 
residual  plots  deviate  from  a  true  straight  line,  a  histogram  of  the  studendzed  residuals  was 
created  (Rgure  6).  Although  the  histogram  was  somewhat  normally  shaped,  the  quantity 
of  residuals  between  0.0  and  0.6  standard  deviations  was  unusually  high.  This  is  due  to 
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the  nature  of  the  sustention  rate  data.  Because  approximately  half  of  the  sustention  rate 
values  are  100%,  it  naturally  increases  the  quantity  of  residuals  within  a  particular 
standard  deviation.  Based  on  the  explanation  provided  by  the  histogram  and  the  W 
Statistic  exceeding  the  established  standard  of  0.9000,  the  assumption  of  normality  is  not 
rejected. 


mik-Shai^  /  Ranlrit  Pbt 


ApprmnMtoWMi-SlMpifO  0.9072  79CMM 


Figure  5.  Wilk-Shapiro/Rankit  Plot  for  Sustention  Rates 


Histogram  for  SUSTRATE 


Figure  6.  Histogram  of  Studentized  Residuals  for  Sustention  Rates 

MulticoUinearity.  During  model  building,  none  of  the  indicators  of 
multicollinearity  were  detected  The  models  did  not  show  a  large  R^  value  and  F-Statistic, 
with  small  individual  t-values,  nor  large  changes  in  the  estimated  coefficients  when  a 
variable  was  added  or  deleted  nor  large  variance  inflation  faemrs.  The  VIFs  for  both 
models,  displayed  in  Tables  33  and  35,  are  well  below  the  maximum  10.0  VIF  often  taken 
as  an  indicator  of  severe  multicollinearity  problems  (Neter  and  others,  1985:392). 

Model  Determination  and  Evaluation.  Because  the  MLR  assumptions  were 
satisfied  the  models  previously  outlined  in  Tables  33  and  35  were  selected  as  the  best 
predictors  for  sustention  rates  and  disposition  times,  respectively.  For  both  iiKxiels,  the  p- 
values,  which  arc  the  observed  significance  levels  of  the  F-statistic,  are  0.0000.  This 
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indicates  that  at  least  one  model  parameter,  p,  is  useful  for  predicting  sustention  rates  or 
disposition  times  at  any  significance  value  greater  than  zero.  This  result  leads  to  the 
conclusitxi  that  a  significant  linear  relationship  exists  between  the  dependent  variables  and 
the  combined  effects  of  the  independent  variables. 

The  predictor  variables  used  in  the  sustention  rate  and/or  disposition  time  models 
are  defined  in  Table  35  and  the  regression  equations  for  each  model  are  also  provided 
below. 


TABLE  35 

PREDICTOR  VARIABLES  USED  IN  MLR  MODELS 


Factor 

Variable 

Definition 

Product  Center 

CENT_ESC 

If  ESC  then  CENT.ESC  =  1 ,  1 

elseCENT_ESC  =  0. 

CO  Disagreement  with  RPA 

DISAGREE 

If  Preneg.  Obj  <  RPA  then  DISAGREE  = 

1.  else  DISAGREE  sO. 

Legal  Complexity 

GRAY 

If  Legally  Complex  then  GRAY  =  1, 
else  GRAY  =  0. 

Identity  of  Prime  Contractor 

KTR_MA 

If  Martin  Marietta  then  KTR.MA  =  1, 
elseKTR_MA  =  0. 

Identity  of  Prime  ContracttH' 

KTR_MC 

If  McDonnell  Douglas  then  KTR  MC  = 
l.elseKTR_MC  =  0. 

Method  of  Disposition 

METHDISP 

If  litigated  then  METHDISP  =  1, 
else  METHDISP  =  0. 

Alleged  Defect  Amount 

RPA_LRG 

If  RPA  a  $1,000,000  then  RPA_LRG  = 
l.elseRPA_LRG  =  0. 

RPA  Revisions 

RPA_REVS 

If  RPA  revised  then  RPA_REVS  =  1, 
eIseRPA_REVS  =  0. 

The  regression  equation  for  sustention  rate  is; 


SUSTENTION  RATE  =  99.2  -  26SXPISAGREE)  -  \2.1{GItAY)  - 

A2.MMETHDISP)  -  2Q.\{RPA_LRG)  -  23JiCENT_ESO  +  21A(KTR_MA) 

The  predictor  variables  are  listed  in  descending  order  of  their  contribution  to 

the  predictive  capability  of  the  sustention  rate  model.  In  other  words,  the  variable 

DISAGREE  contributes  more  to  the  model's  predictive  capability  than  GRAY,  and  GRAY 
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contributes  more  than  METHDISP,  and  so  on.  The  coefficient  of  determination,  R2,  for 
the  entire  regression  model  is  0.7339.  This  means  that  the  combination  of  the  six  selected 
predictor  variables  explains  73  4%  of  the  variability  in  sustention  rates. 

The  coefficients  associated  with  each  predictor  variable  indicate  the  magnitude  by 
which  the  dependent  variable,  SUSTENTION  RATE,  is  affected.  For  example,  if  the  case 
is  expected  to  go  to  litigation,  the  predicted  sustention  rate  will  decrease  by  42.4%;  if  the 
prime  contractor  is  Martin  Marietta,  the  predicted  rate  will  increase  by  21.4%.  Therefore, 
METHDISP  has  a  larger,  but  opposite  effect  on  SUSTENTION  RATE  than  does 
KTR_MA.  Because  all  the  predictor  variables  are  indicator  variables,  the  model  can  be 
used  in  the  following  manner;  Step  1 :  Assume  an  initial  sustention  rate  of  99.2%;  Step  2: 
If  the  predictor  variable  is  true,  subtract  or  add  the  variable's  coefficient  amount;  Step  3: 
Repeat  Step  2  until  no  more  predictor  variables  are  true.  The  end  result  is  a  final 
predicted  sustention  rate  for  the  particular  case  being  analyzed. 

The  regression  equation  for  disposition  time,  as  outlined  in  Table  23,  is: 

LOG  DISPOSITION  TIME  =  2.54  +  0.25{GRAY)  +  0.33iRPA_REVS)  - 
0.47(jm?_AfO  +  0.3S(RPA^LRG) 

As  with  the  sustention  rate  regression  equation,  the  predictor  variables  for 
(logarithmic)  disposition  times  are  listed  in  descending  order  of  their  contribution  to  the 
predictive  capability  of  the  sustention  rate  model.  Therefore,  GRAY  contributes  more 
than  RPA_REVS,  RPA_REVS  more  than  KTR_MU,  and  KTR_MC  more  than 
RPA_LRG.  The  coefficient  of  determination,  r2,  for  the  entire  regression  model  is 
0.4S49.  This  means  that  the  combination  of  the  four  selected  predictor  variables  explains 
48.5%  of  the  variability  in  disposition  times. 

The  coefficients  associated  with  each  predictor  variable  indicate  the  magnitude  by 
which  the  dependent  variable,  LOG  DISPOSITION  TIME,  is  affected.  Thus,  McDonnell 
Douglas'  cases  have  the  larger  impact  on  decreasing  disposition  times;  cases  with  RPAs 
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over  $1M  have  the  largest  impact  on  increasing  disposition  times.  Because  a  logarithmic 
transfonnation  of  disposition  dmes  was  necessary  to  equalize  the  variances,  the  coefficient 
values  and,  thus,  the  predicted  times  will  be  in  logarithmic  form.  As  such,  a  re- 
transformation  (anti-log)  will  need  to  be  accomplished  in  order  to  determine  the  predicted 
disposition  dme. 

Model  Validation 

One  third  of  the  original  data  that  was  collected  was  set  aside  to  investigate  the 
predictive  capability  of  the  selected  models.  The  non-fitted  data  were  entered  into  the 
regression  equations  and  a  predictive  R2  was  calculated  for  each  model.  The  results  are 
provided  below  in  Table  36. 


TABLE  36 

COMPARISON  OF  FITTED  R2  AND  PREDICTIVE  R2 


Model 

Fitted  R2 

Predictive  R2 

Difference  | 

Sustention  Rates 

0.7339 

0.5977 

0.1362 

Disposition  Times 

0.4849 

0.3292 

0.1557  1 

The  results  indicate  a  loss  of  predictive  capability  for  both  models  when  non-fitted 
data  were  used.  This  result  is  not  unexpected  given  the  nature  of  model  building. 
Approximately  60%  and  33%  of  the  variation  in  sustention  rates  and  disposition  times, 
respectively,  were  explained  by  the  models  when  unfitted  data  were  used. 
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V.  SUMMARY  AND  CONCLUSIONS 


Summary 

Defective  pricing  occurs  when  contractors  fail  to  disclose  current,  accurate,  and 
con^lete  cost  or  pricing  data  in  their  proposals.  Failure  to  submit  valid  cost  or  pricing 
data  entitles  the  government  to  a  refund  in  the  amount  of  the  overpayment  With  the 
current  backlog  of  overage  defective  pricing  cases  and  the  continuing  decline  in  sustention 
rates,  a  better  understanding  of  the  underlying  factors  inhibnting  or  fostering  timely  and 
successful  recoupment  of  defective  pricing  funds  is  needed.  The  purpose  of  ■  ns  research 
was  to  identify  the  factors  which  significantly  affect  sustention  rates  and  disposition  times 
in  defective  pricing  cases,  and  determine  if  valid  models  could  be  developed  to  predict 
both  rates  and  times. 

Research  Objectives.  The  specific  objectives  of  this  research  were  to: 

1.  Identify  factors  that  are  presumed  to  affect  sustention  rates  of  defective  pricing 
cases. 

2.  Identify  factors  that  are  presiuned  to  affect  disposition  times  of  defective  pricing 
cases. 

3.  Determine  which  factors  identified  in  Objectives  1  and  2  explain  the  greatest 
variation  in  sustention  rates  and  disposititm  times. 

4.  Develop  and  validate  a  model  to  predict  sustention  rates  using  factors  identified 
in  Objective  3. 

5.  Develop  and  validate  a  model  m  predict  disposition  times  using  factors  identified 
in  Objective  3. 

Metfaodolofv  and  .Siirnmarv  Results.  A  summary  of  the  methodology  and  results  for 
each  objective  are  provided  below. 
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Research  Objectives  1  and  2.  On  the  basis  of  a  literature  review,  personal 
experience,  and  discussions  with  defective  pricing  experts,  a  list  of  potential  factors 
affecting  sustention  rates  and  disposition  times  was  compiled.  To  validate  this  initial  set  of 
factors  and  identify  important  additional  factors,  twenty-four  interviews  were  conducted 
with  defective  pricing  experts.  The  resulting  list  of  factors,  which  the  interviewees 
presumed  to  significantly  ajffect  sustention  rates  and  disposition  times,  is  shown  in  the  first 
two  columns  of  Table  37. 

Research  Objective  3.  The  variables  found  to  be  significant  during  the 
interview  process  were  screened  for  quantitative  analysis.  Several  factors,  such  as 
expertise  of  participants,  workload,  and  priority,  were  excluded  from  further  analysis  due 
to  the  difficulty  in  collecting  data  or  the  inability  to  determine  what  type  of  data  should  be 
collected  to  accurately  measure  the  factor.  For  the  factors  which  lent  themselves  to 
quantification,  data  were  collected  on  defective  pricing  cases  resolved  between  1  March 
1991  and  31  March  1994  at  ASC,  ESC,  and  SMC.  A  total  of  1 18  cases  were  analyzed. 

The  interview  results  indicated  that  offsets  and  the  identity  of  the  prime  contractor 
were  considered  significant  for  disposition  times,  but  not  sustention  rates.  Because  data 
were  collected  on  those  variables,  the  statistical  effect  of  offsets  and  the  identity  of  the 
prime  contractor  on  sustention  rates  was  also  assessed. 

For  each  of  the  factors  meriting  statistical  analysis,  a  one-way  analysis  of  variance 
model  was  built  to  determine  which  fa>.  tors  contributed  most  to  explaining  sustention  rates 
and  disposition  times.  The  results  were  evaluated  to  ascertain  which  factors  showed  a 
statistically  significant  difference  in  the  factor  level  means  at  the  95%  confidence  level. 

The  statistical  evaluation  of  the  factors,  compared  to  the  expert^'  expectations,  is 
presented  in  colunm  three  of  Table  37. 
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TABLE  37 


RESULTS  FOR  RESEARCH  OBJECTIVES  1, 2,  AND  3 


Experts'  Assessment 
of  E^ecton  Rates 
and  Times 


Alleged  Defect  Amount 


Cost  Element 


Contractor  Level 


ment  with  RPA 


Number  of  Issues 


Offsets 


Method  of  Disposition 


Fraud  Investigation 


RPA  Revisions 


Disposition  Time 


Identity  of  Prime  Contractor 


Expertise  of  Negotiators 


Product  Center 


Workload 


Priori 


Interest  Assessed 


Audit  Characteristics 


Assist  Audits 


Documentation 


se  of  Participants 


Personnel  Turnover 


t  Attention 


lEB 


ANOVA  Results  -  Statistical 
Confirmation  of  Experts' 
Expectations 


Confirmed 


Not  Confirmed 


Confirmed  for  Rates  Onl 


Confirmed  for  Rates  Onl 


Confirmed 


Confirmed 


Not  Confirmed,  but  Statistically 
Significant  for  Rates 


Confirmed 


Not  Confirmed 


Confirmed  for  Times  Onl 


Confirmed 


Confirmed  -  Also  Found  to  be 
Statistically  Significant  for  Rates 


Not  Evaluated 


Confirmed 


Not  Evaluated 


Not  Evaluated 


Confirmed 


Not  Evaluated 


Not  Evaluated 


Not  Evaluated 


Not  Evaluated 


Not  Evaluated 


Not  Evaluated 


Research  Objectives  4  and  5.  The  factt»s  found  to  be  statistically  significant 
using  analysis  of  variance  became  the  initial  set  of  predictor  variables  ccmsidered  for  model 
building.  Disposition  Time  was  excluded  as  a  potential  fnedictor  variable  because  the 
value  of  Disposition  Time  is  unknown  until  the  case  has  been  settled.  Therefore,  the 
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indq)eixlent  variable,  Disposition  Time,  is  not  useful  for  prediction.  Using  multiple  linear 
regression  techniques,  the  models  shown  below  were  selected  as  the  best  predictors  for 
sustention  rates  and  disposition  times.  The  ch(»en  models  explain  approximately  73.4% 
and  48.5%  of  the  variability  for  sustention  rates  and  disposition  times,  respectively. 

SUSTENTION  RATE  =  99.2  -  26.0(DISAGREE)  -  12 J (GRAY)  - 

42A(METHDISP)  -  20 A(RPA_LRG)  -  23J(CENT_ESC)  +  21  A{KTR_MA) 

LOG  DISPOSITION  TIME  =  2.54  +  0.25(GRAy)  +  0.33(RPA_REVS)  - 
OAliKTR_MC)  +  Q.3SiRPA_LRG) 

The  predictor  variables  are  listed  in  descending  order  of  their  contribution  to  the 
predictive  capability  of  the  models  and  are  defined  in  Table  38.  In  the  disposition  model, 
for  example,  GRAY  contributes  more  than  RPA_REVS,  RPA_REVS  more  than 
KTR_MC,  and  KTR_MC  more  than  RPA_LRG.  TTie  coefficients  associated  with  each 
predictor  vaiiable  indicate  the  magnitude  by  which  the  dependent  variables, 
SUSTENTION  RATES  and  LOG  DISPOSITION  TIMES,  are  affected.  Thus, 
METHDISP  has  the  largest  impact  on  decreasing  sustention  rates  and  RPA_LRG  has  the 
largest  impact  on  increasing  disposition  times. 
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TABLE  38 


PREDICTOR  VARIABLES  FOR  REGRESSION  MODELS 


Factor 

Predictor  Variable 

Definition 

Product  Center 

CENT.ESC 

If  ESC  then  CENT_ESC  =  1, 
else  C:ENT_ESC  =  0. 

CO  Disagreement 
withRPA 

DISAGREE 

If  Preneg  Obj  <  RPA  then  DISAGREE  = 
Lelse  DISAGREE  =  0. 

Legal  (Complexity 

GRAY 

If  Legally  Complex  then  GRAY  =  1, 
else  GRAY  =  0. 

Identity  of  Prime 
Contractor 

KTR_MA 

If  Martin  Marietta  then  KTR_MA  =  1, 
eIseKTR_MA  =  0. 

Identity  of  Prime 
Contractor 

KTR_MC 

If  McDonnell  Douglas  then  KTR_MC  = 
l,clseKTR_MC=0. 

Method  of 
Disposition 

METHDISP 

If  litigated  then  METHDISP  =  1, 
else  METHDISP  =  0. 

Alleged  Defect 
Amount 

RPA_LRG 

If  RPA  5  $1,000,000  then  RPA.LRG  = 
l,elseRPA_LRG  =  0. 

RPA  Revisions 

RPA.REVS 

If  RPA  revised  then  RPA_REVS  =  1 , 
elseRPA_REVS=0. 

Because  ail  the  predictor  variables  are  indicator  variables,  the  models  can  be  used  in 
the  following  manner.  Step  1:  Assume  an  initial  sustention  rate  of  99.2%;  Step  2:  If  the 
predicttv  variable  is  true,  subtract  or  add  the  variable’s  coefficient  amount;  Step  3:  Repeat 
Step  2  until  no  more  predictor  variables  are  true.  The  end  result  is  a  final  predicted 
sustention  rate  for  the  particular  case  being  analyzed.  The  same  methodology  applies  for 
disposition  times,  but  because  the  results  for  the  disposition  time  model  are  in  logarithmic 
form,  a  re-transformation  (anti-log)  needs  to  be  accomplished  to  obtain  the  predicted 
disposition  time. 

One  third  of  the  original  data  collected  was  set  aside  to  investigate  the  predictive 
capability  of  the  selected  models.  The  non-fitted  data  were  entered  into  the  selected 
regression  equations  and  a  predictive  R^  was  calculated  for  each  model.  Approximately 
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60.0%  and  32.9%  of  the  variation  in  sustention  rates  and  disposition  times,  respectively,  is 
explained  by  the  models  when  unfitted  data  are  used. 

Conclusions  and  Findings  for  Sustention  Rates 

Based  on  the  research  conducted  for  this  thesis,  the  factors  which  provide  the  best 
overall  explanation  of  sustention  rates  are  discussed  below.  These  factors  were  identified 
as  statistically  significant  by  analysis  of  variance  techniques.  Factors  suggested  by  the 
experts  for  which  data  were  not  collected  may  be  important  in  explaining  rates  and  times, 
but,  absent  statistical  evaluation,  no  conclusions  can  be  drawn.  It  is  important  to  note  that 
the  follovring  discussion  pertains  to  each  factra*  considered  independently,  not  in 
combination  with  the  other  factors.  The  regression  models  provide  the  best  indication  of 
the  effect  these  variables  have  in  combination  with  one  another. 

Alleged  Defect  Amount.  Sustention  rates  decrease  as  the  size  of  the  RPA 
increases.  Cases  with  an  RPA  less  than  $1,000,000  averaged  89%;  cases  exceeding 
$1,000,000  averaged  only  56.6%.  It  is  believed  that  the  magnitude  of  the  RPA  affects  the 
seriousness  with  which  a  case  is  pursued  by  the  contractor.  This  increased  contractor 
resistance  is  a  precursor  to  lower  settlement  amounts,  in  part  due  to  the  government's 
desire  to  settle,  not  litigate,  defective  pricing  cases. 

Contractor  Level.  Sustention  rates  for  cases  involving  defective  pricing  at  the 
prime  contractor  level  are  an  average  of  24%  higher  than  those  at  the  subcontractor  level. 
In  subcontractor  defective  pricing  there  are  two  interested  parties,  the  prime  and  the 
subcontractor.  Both  parties  have  a  financial  stake  in  the  outcome  of  the  case;  the 
subcontracts  pays  back  the  amount  of  overpayment  and  the  prime  pays  back  the  burden 
anKxmt  As  such,  both  parties  are  likely  to  provide  more  resistance  to  the  defective 
pricing  allegations.  Increased  contracts  resistance  is  a  ptecurss  for  lower  sustention 
rates. 
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Contracting  Officer  Disagreement  with  RPA.  Disagreement  with  the  audit 
report,  measured  by  whether  or  not  the  negotiation  objective  is  less  than  the  final  RPA, 
has  a  very  strong  effect  on  sustention  rates.  Inmitively,  CO  disagreement  with  the  RPA 
leads  to  lower  sustention  rates.  In  fact,  the  data  show  that  the  sustention  rate  is  almost 
37%  lower  in  cases  involving  disagreement  It  is  difficult  to  criticize  lower  sustention 
rates  based  on  this  factor  alone,  however,  because  it  reflects  disagreement  within  the 
government  on  how  much  the  contractor  is  required  to  repay.  To  criticize  the  contracting 
officer  in  such  situations  is  equivalent  to  assuming  that  the  auditor's  position  was  correct 
The  presence  of  disagreement  within  the  government  highlights  the  difficulties  in 
determining  whether  or  not  a  contractor  is  liable  for  defective  pricing,  and,  if  so,  how 
much  is  to  be  repaid. 

Number  of  Issues.  Sustention  rates  are  an  average  of  14%  lower  in  cases  involving 
multiple  issues  than  those  involving  (xily  one  issi».  It  is  believed  that  the  consolidation  of 
multiple  issues  into  one  audit  report  increases  the  complexity  of  the  case,  making  it  tiK>re 
difficult  for  the  government  to  negotiate  a  full  settlement 

Legal  Complexity.  Sustention  rates  are  much  lower  for  defective  pricing  cases 
involving  legally  complex  issues;  the  average  sustention  rate  for  complex  cases  was  62.9% 
as  compared  to  95.5%  fix'  non-complex  cases.  The  cases  categorized  as  legally  complex 
involved  contractor  claims  that  the  data  in  question  were,  in  fact,  appropriately  disclosed, 
that  the  data  did  not  meet  the  definition  of  cost  or  pricing  data,  or  that  the  auditor's 
rationale  for  calculating  the  allegation  amount  was  erroneous.  Inconsistent  case  law  on 
these  issues  paired  with  poor  or  missing  government  documentation  made  the 
government's  position  difficult  to  defend.  Faced  with  high  litigation  risk,  the  government 
negotiator  will  often  settle  the  case  for  less  than  the  full  RPA.  If  the  case  is  litigated,  the 
sustention  rate,  as  discussed  below,  is  likely  to  be  even  lower. 
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Cases  categorized  as  not  complex  typically  involved  contractor  failure  to  submit 
current  rates  or  updated  vendor  quotes.  These  cases  lack  the  interpretative  ambiguities 
discussed  above  and  are  difficult  for  the  contractor  to  refute.  The  auditor  usually  has 
evidence  of  the  updated  quote  and  can  easily  verify  whether  or  not  it  was  submitted. 
Therefore,  the  RPA  is  normally  fully  sustained. 

Ofbets.  The  presence  of  offset  proposals  and  the  government's  evaluation  of  the 
proposals  have  a  strong  effect  on  sustention  rates.  Cases  in  which  offsets  are  not  accepted 
have  the  highest  sustention  rates  (99%).  This  result  is  contrary  to  the  expectation  that 
refused  offsets  would  increase  the  amount  owed  by  the  contractor,  thereby  increasing  his 
resistance  and  decreasing  the  sustention  rate.  In  some  of  the  cases  which  had  rejected 
offsets,  the  government  had  a  very  sound  case.  Because  the  case  was  so  strong,  the 
contractor  resorted  to  submitting  unsubstantiated  offsets  in  an  attempt  to  reduce  the  debt. 
These  offsets  were  then  disallowed,  and  the  contractor  paid  the  full  defect  amount, 
resulting  in  a  high  sustention  rate. 

Fully  accepted  offsets  have  the  next  highest  sustention  rates  (94%).  Offsets 
represent  an  opportunity  for  the  contractor  to  reduce  the  amount  owed  to  the  government. 
As  such,  accepted  offsets  may  lead  to  reduced  resistance  from  the  contractor  and  higher 
sustention  rates.  Cases  in  which  no  offsets  were  proposed  have  the  second  lowest 
sustention  rates  (81%).  It  is  presumed  that  in  these  cases,  there  were  no  valid  offsets  to 
propose  so  the  contractor  could  not  reduce  the  amount  owed  to  the  government.  Because 
the  size  of  the  RPA  had  not  diminished,  the  contractors  increased  their  resistance  to 
allegation,  thereby  lowering  the  sustention  rate. 

The  lowest  sustention  rates  were  cases  in  which  the  offsets  were  only  partially 
accepted  by  the  contracting  officer  (74%).  This  low  rate  may  be  explained  by  the  fact 
that,  in  some  of  these  cases,  the  contracting  officer  partially  accepted  an  offset  that  the 
auditOT  rejected.  When  an  auditcMr  rejects  an  offset,  his  RPA  remains  unchanged.  When  a 
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contracting  officer  partially  accepts  an  offset  rejected  by  the  auditor,  there  will  be  a  lower 
sustention  rate,  by  definition. 

Method  of  Disposition.  Litigated  cases  result  in  much  lower  sustention  rates  than 
do  cases  settled  without  litigation.  On  average,  the  sustention  rate  for  litigated  cases  is 
23.3%;  for  cases  not  litigated,  it  is  84.6%.  Cases  that  are  settled  by  the  boards  or  courts 
usually  involve  complex  interpretative  issues.  Neither  the  government  nor  the  contractor 
clearly  prevails,  so  the  courts  often  split  the  difference. 

Disposition  Time.  Longer  disposition  times  are  associated  with  lower  sustention 
rates.  To  illustrate,  the  average  sustention  rate  for  cases  settled  within  one  year  is  94%. 

In  contrast,  cases  taking  more  than  three  years  to  resolve  average  only  75%.  Contributing 
factors  include  the  increased  probability  of  lost  documentation  and  personnel  mmover  as 
time  passes.  These  factors  weaken  the  government's  ability  to  meet  the  burden  of  proof  in 
a  defective  pricing  case  and  lead  to  lower  sustention  rates.  As  such,  disposition  times  and 
sustention  rates  are  directly  related.  Management  emphasis  on  the  prompt  resolution  of 
defective  pricing  cases  satisfies  regulatory  requirements,  and,  more  importantly,  is 
associated  with  higher  sustention  rates. 

Identity  of  the  Prime  Contractor.  The  specific  contractor  involved  in  a  defective 
pricing  case  has  an  effect  on  the  sustention  rate.  For  the  data  analyzed,  sustention  rates 
ranged  from  60%  to  102%,  depending  on  the  contractor  involved.  It  is  believed  that 
corporate  culture,  attitudes  toward  defective  pricing,  and  experience  with  defective  pricing 
affect  how  companies  handle  defective  pricing  allegations.  This  in  turn  has  a 
demonstrated  effect  on  sustention  rates. 

Product  Center.  The  product  center  resolving  the  defective  pricing  case  affects 
sustention  rates,  and,  on  average,  ESC  cases  result  in  lower  sustention  rates  (63.9%)  than 
ASC  (86.7%)  and  SMC  (80.7%)  cases.  Because  no  statistically  significant  difference  in 
rates  was  detected  between  SMC  and  ASC,  it  cannot  be  concluded  that  organizational 
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structure  (whether  cases  are  handled  on  a  centralized  or  decentralized  basis)  is  a 
contributing  factor.  The  results  imply  that  the  differences  in  sustention  rates  are  driven  by 
factors  related  to  the  individual  centers,  not  organizational  structures.  Such  factors 
include  management  attention,  priority  levels,  and  the  nature  of  the  items  purchased, 
including  the  concentration  ratios  of  the  industries  involved  as  well  as  the  specific 
contractors. 

Interest  Assessed.  The  data  show  that  sustention  rates  are  an  average  of  15% 
lower  in  cases  where  interest  was  assessed.  During  the  interviews  it  was  suggested  that 
this  results  6om  defective  pricing  amounts  being  rolled  together  with  interest  charges. 
Oftentimes  the  ctxitractor  Avill  agree  to  repay  a  certain  total  sum  which  includes  both  the 
defective  pricing  repayment  and  the  interest  charge.  This  bottom-line  proposal  is  rarely 
more  than  the  OTiginal  allegation  amount  For  the  government  to  assess  interest,  the 
interest  amount  must,  in  effect  be  subtracted  ffom  the  amount  sustained.  The  end  result 
is  that  roughly  the  same  amount  of  money  is  recovered,  but  in  cases  assessing  interest  the 
sustention  rates  are  lower. 

Conclusions  and  Findings  for  Disposition  Times 

The  factors  which  provide  the  best  overall  explanation  of  disposition  times  are 
discussed  below,  along  with  any  general  findings  associated  with  these  factors.  As  for 
sustention  rates,  the  factors  identified  as  statistically  significant  by  analysis  of  variance 
techniques  are  included.  In  addition,  two  factms  which  were  not  statistically  significant 
offsets  and  fraud  investigation,  merit  additional  discussion  and  are  also  included. 

Alleged  Defect  Amount.  Disposition  time  increases  as  the  magnitude  of  the  RPA 
increases.  Cases  with  an  RPA  exceeding  $1,000,000  took  an  average  of  1397  days;  cases 
less  than  $1,000,000  averaged  only  840  days.  It  is  believed  that  the  magnitude  of  the  RPA 
affects  the  seriousness  with  which  a  case  is  pursued  by  the  contractor.  The  greater  the 
RPA,  the  more  aggressive  the  contractor  becomes  in  refuting  elements  of  the  defective 
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pricing  audit  With  each  contractor  rebuttal,  additional  time  is  expended  by  the 
contracting  and  audit  agencies  to  evaluate  the  validity  of  the  contractor's  claims.  The 
impact  of  this  reiterative  exchange  is  longer  disposition  times. 

Number  of  Issues.  Cases  involving  multiple  issues  took  an  average  of  535  days 
longer  to  settle  than  cases  involving  only  one  issue.  This  is  likely  due  to  the  fact  that  each 
issue  becomes,  in  effect,  a  separate  negotiation. 

Legal  Complexity.  Disposition  times  are  much  longer  for  cases  which  are  more 
legally  complex.  In  this  research,  complex  cases  took  an  average  of  twice  as  long  to  settle 
as  non-complex  cases. 

Offsets.  Contractor  proposed  offsets  increase  the  time  it  takes  to  resolve  a 
defective  pricing  case.  Cases  in  which  no  offsets  were  proposed  mok  the  least  amount  of 
time  (846  days),  offsets  proposed  and  fully  accepted  by  the  contracting  officer  were  next 
(958  days),  followed  by  partially  accepted  offsets  (1 199  days),  and,  finally,  offsets  rejected 
in  full  (1305  days). 

While  the  analysis  of  variance  did  not  show  offsets  as  statistically  significant  for 
disposition  times,  the  experts  and  researchers  believe  that  offsets  do  affect  times.  The 
disconnect  between  the  statistical  test  results  and  expert  perceptions  may  have  to  do  with 
the  timing  of  submitted  offsets.  If  offsets  are  submitted  before  the  initial  audit  is  released, 
disposition  times  are  not  affected  because  disposition  time  begins  when  the  initial  audit  is 
issued.  Offsets  submitted  immediately  after  the  release  of  the  initial  audit  will  impact 
disposition  times,  but  to  a  lesser  extent  than  offsets  proposed  much  later  in  the  resolution 
process.  This  stems  from  the  fact  that  the  auditor  may  be  reviewing  additional  comments 
by  the  contractor  and  contracting  officer  early  in  the  process  rather  than  later.  Therefore, 
offsets  proposed  earlier  in  the  process  lessen  the  damage  to  disposition  times. 
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Method  of  Disposition.  Cases  which  are  litigated  take,  on  average,  more  than 
twice  as  long  to  settle  than  cases  resolved  by  negotiations,  contracting  officer  final 
decision,  or  out-of-court  settlements. 

RPA  Revisions.  The  data  showed  that  RPA  revisions  add  an  average  of  almost  500 
days  to  the  disposition  time.  After  the  initial  audit  is  released,  fact-finding  sessions, 
discussions  with  the  cognizant  contracting  officer,  and  consideration  of  contractor 
responses  to  the  allegation  of  defective  pricing  often  occur.  This  additional  information 
frequently  leads  to  reconsideration  of  die  case  by  the  auditor  and  a  revision  to  the  RPA. 
While  these  revisions  are  necessary,  the  evidence  clearly  shows  that  they  lengthen 
disposition  times. 

Identity  of  Prime  Contractor.  Who  the  contractor  is  has  considerable  bearing  on 
the  time  it  takes  to  resolve  a  case.  For  the  data  analyzed,  average  disposition  times  ranged 
from  418  to  1608  days,  depending  on  the  contractor  involved.  Corporate  policies, 
attitudes,  and  willingness  to  settle  defective  pricing  cases  are  contributing  elements  toward 
an  expedient  or  lengthy  settlement  of  a  case. 

Product  Center.  The  difference  in  disposition  times  is  more  affected  by  the  identity 
of  the  product  center  than  the  type  of  organizational  structure  used  to  resolve  defective 
pricing  cases.  Cases  resolved  by  ESC  (decentralized)  took,  on  average,  twice  as  long  as 
cases  resolved  by  SMC  (decentralized)  or  ASC  (centralized). 

Interest  Assessed.  Disposition  times  are  an  average  of  290  days  shorter  for  cases  in 
which  interest  was  assessed.  The  knowledge  that  the  amount  owed  to  the  government 
increases  over  time  appears  to  motivate  contractors  to  settle  cases  more  quickly.  The 
TINA  amendment  mandating  interest  payments  for  defective  pricing  cases  appears  to  be 
effective  in  decreasing  disposition  times. 

Fraud.  While  the  analysis  of  variance  did  not  show  offsets  as  statistically  significant 
for  disposition  times  due,  in  part,  to  the  low  number  of  data  points,  the  experts  and 
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researchers  believe  that  offsets  do  affect  times.  The  descriptive  statistics  for  this  research 
suppOTt  this  belief;  cases  investigated  for  fraud  added  an  average  of  436  days  to 
disposition  times.  Of  the  seven  cases  investigated,  six  lacked  criminal  evidence  and  were 
returned  to  the  contracting  agency  for  final  settlement.  Some  of  these  cases  were 
unnecessarily  lengthened  by  the  fact  that  the  investigation  had  been  closed  for  several 
months  before  the  contracting  officer  was  notified. 

Additional  Conclusions  and  Findings 

This  research  provided  additional  insight  into  factors  which  were  believed  to 
significantly  effect  sustention  rates  and  disposition  times,  but  the  analysis  of  variance  did 
not  confirm  the  expens'  beliefs.  A  short  synopsis  of  those  factors  not  confirmed  through 
statistic  analysis  is  provided  below,  as  well  as  some  additional  findings  associated  with  this 
research. 

1.  Experts  believed  that  the  cost  element  in  which  the  defect  occurred  had  a 
moderate  or  greater  effect  on  sustention  rates  and  disposition  times.  This  belief  was  not 
substantiated  by  the  analysis  of  variance  results.  The  researchers  believe  that  the  type  of 
defect,  such  as  failure  to  submit  updated  rates  or  adequately  disclose  a  management 
decision,  is  the  contributing  factor,  not  where  the  defect  was  found.  Experts  also  believed 
that  the  contractor  level,  prime  or  subcontract,  affected  both  rates  and  times.  The 
ANOVA  results  showed  that  the  level  substantially  affected  rates,  but  not  times.  The 
most  surprising  result  was  in  the  area  of  offsets.  Experts  believed  that  offsets  had  a  strong 
effect  on  times,  but  not  rates.  The  analysis  of  variance  results  found  the  opposite  to  be 
true.  The  reasons  why  this  may  have  occurred  were  discussed  in  the  previous  sections, 

2.  In  her  1988  thesis,  Volpe  stated  that  the  time  it  took  to  process  a  case  was  not 
related  to  the  alleged  defect  amount.  Instead,  she  found  a  correlation  between  processing 
time  and  the  audited  contract  amount-the  greater  the  amount,  the  longer  it  took  to  settle 
the  case.  The  researchers  of  this  thesis  found  the  opposite  to  be  true;  disposition  times  are 


related  to  the  alleged  defect  amount  (RPA),  and  not  the  audited  contract  amount. 

Because  the  correlation  between  the  audited  contract  amount  and  disposition  times  was  so 
low,  -0.0520,  Volpe's  finding  was  not  substantiated  by  this  research.  Interview  comments 
also  support  the  researchers'  statistical  findings.  The  experts  believe  that  the  value  of  the 
RPA  is  a  more  critical  factor  than  either  the  audit  contract  amount  or  the  ratio  of  the  RPA 
to  the  audited  contract  amount. 

3.  The  multiple  linear  regression  model  for  sustention  rates  had  a  much  better 
predictive  capability  than  the  model  for  disposition  times.  The  researchers  believe  the 
omitted  factors,  such  as  priority  and  expertise  of  defective  pricing  participants,  are  key 
predictors  of  disposition  times,  but  less  influential  for  sustention  rates.  This  fact 
en^hasizes  the  tinding  that  factors  affecting  sustention  rates  are  substantially  different 
than  factOTS  affecting  disposition  times.  To  improve  the  predictive  capability  of  a 
dispositicm  time  model,  participant  expertise,  workload,  priority,  personnel  turnover,  and 
management  attention  need  to  be  examined. 

4.  For  the  1 18  cases  analyzed,  sustention  rates  averaged  83%  and  disposition  times, 
944  days  or  just  over  2Vi  years.  The  118  cases  did  not  include  29  cases  which  had  a 
reported  $0  sustention  amount,  but  a  non-zero  RPA.  Including  these  29  cases  drops  the 
overall  sustention  rate  to  67%,  a  figure  which  represents  the  DoD's  method  of  officially 
reporting  sustention  rates.  The  zero  dollar  cases  were  those  in  which  the  contracting 
activity  disagreed  completely  with  the  auditor's  position  and  did  not  pursue  the  defective 
pricing  allegation.  ASC  disagreed  completely  with  the  auditor's  position  approximately 
10%  of  the  time;  ESC,  18%;  and  SMC,  49%.  With  the  dramatic  difference  in  the  number 
of  cases  not  pursued  by  SMC,  the  researchers  believe  that  contracting  officers  may  have 
inappropriately  closed  cases  which  were,  in  fact,  legitimate.  Taking  these  cases  into 
consideration,  sustention  rates  fall  from  86.7%  to  80.1%  for  ASC;  from  63.9%  to  55.9% 
for  ESC;  and  from  80.7%  to  39.3%  for  SMC.  Direct  measurement  of  disposition  times  of 
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these  cases  was  not  accomplished.  Therefore,  the  impact  to  disposition  times  is  unknown, 
but  given  partial  data  provided  in  PARS,  the  researchers  believe  disposition  times  would 
increase  if  these  cases  were  included.  This  stems  fiom  the  fact  that  these  cases  often 
involve  disagreements  between  the  contracting  officer  and  auditor  on  the  applicability  of 
TINA,  and  considerable  time  is  spent  by  the  contracting  officer  working  with  the  auditor 
to  resolve  their  differences. 

Recommendations  for  Improving  Sustention  Rates 

This  research  has  provided  insight  into  the  factors  which  affect  sustention  rates.  For 
some  of  these  factors,  awareness  of  the  effect  is  valuable,  but  no  action  can  be  taken  to 
increase  rates.  For  other  factors,  changes  can  be  made  by  the  contractor,  contracting 
activity,  and  audit  agency  to  improve  rates.  The  recoimnended  changes  are  provided 
below: 

1.  The  research  showed  that  audit  reports  containing  multiple  issues  result  in  lower 
sustention  rates  than  audits  widi  only  one  issue.  During  the  interviews,  it  was 
recommended  that  when  several  complex  issues  are  involved,  separate  audit  reports 
should  be  issued  for  each  issue.  While  this  may  decrease  the  complexity  of  each 
negotiation  and  increase  sustention  rates,  the  potential  increase  in  audit  manpower  and 
total  disposition  time  must  also  be  considered. 

2.  Because  sustention  rates  are  so  low  for  litigated  cases,  contracting  officers 
shraild  carefully  weigh  the  benefits  of  litigation  and  the  cost  of  a  less  than  optimal 
negotiated  settlement.  In  most  cases,  compromising  in  a  negotiated  settlement  will  result 
in  a  higher  sustention  rate  than  litigation.  Litigation  may  still  be  appropriate,  however,  to 
establish  precedence  on  a  new  or  unresolved  legal  issue. 

3.  Sustention  rates  decline  steadily  as  the  age  of  the  case  increases.  Therefore, 
additional  emphasis  on  prompt  resolution  of  defective  pricing  cases  is  required. 
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4.  The  research  found  that  sustention  rates  are  lower  in  cases  where  the  contracting 
officer  disagrees  with  the  audit  report  Because  of  the  difficult  issues  involved,  some 
disagreement  within  the  government  is  inevitable.  In  some  cases,  however,  the  apparent 
disagreement  is  caused  by  the  auditor's  reluctance  to  revise  his  RPA.  Once  an  audit  report 
has  been  formally  issued,  the  auditor  is  committed  to  that  position  and  may  hesitate  to 
revise  his  position.  Increased  communication  between  the  auditor,  the  contracting  officer, 
and  the  contractor  prior  to  audit  issuance  can  reduce  this  problem.  Doing  so  allows 
disagreements  to  be  identified  and  reconciled  while  the  audit  is  still  in  draft  form,  and 
therefore  avoids  formal  changes  to  official  positions.  This  reduces  the  need  for  future 
revisions  and  the  level  of  disagreement  between  the  auditor  and  the  contracting  officer. 

In  other  cases,  disagreement  and  eventual  closure  of  cases  with  a  zero  sustention 
rate  are  the  result  of  an  inappropriate  contracting  officer  decision.  Cases  closed  by  the 
contracting  officer  may  have  been,  in  fact,  legitimate  cases.  To  prevent  inappropriate 
closures  of  defective  pricing  cases,  the  researchers  recommend  that  the  defective  pricing 
focal  point,  or  some  other  appropriate  individual,  review  the  contracting  officer’s  rationale 
for  not  pursuing  a  defective  pricing  case. 

5.  A  policy  for  the  consistent  recording  of  RPA  revisions  should  be  established.  In 
the  cases  reviewed,  auditors  revised  their  RPAs  several  different  ways,  including  formal 
audit  supplements,  letters  to  the  contracting  officer,  and  telephone  or  personal 
conversations.  Some  of  these  revisions  were  reflected  in  the  PARS  data  system,  others 
were  not.  Since  the  sustention  rate  is  computed  directly  from  the  final  RPA,  clear  policy 
on  the  formality  required  for  RPA  revisions  is  needed  to  prevent  distortion  of  the 
statistics. 

6.  Another  administrative  recommendation  involves  the  recording  of  offsets. 
Inconsistent  recording  of  offsets  by  the  contracting  activity  has  lead  to  distorted  sustention 
rate  statistics.  In  many  of  the  cases  reviewed,  the  auditor  reduced  the  RPA  by  the  amount 
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of  the  accepted  offset.  In  others,  the  RPA  remained  unchanged,  and  the  offset  was 
recognized  separately  as  a  reduction  in  the  amount  to  be  repaid  to  the  government.  While 
either  accounting  method  is  acceptable,  contracting  agencies  must  be  sure  to  minor  the 
auditor's  practice  when  reporting  the  amount  sustained. 

Recommendations  for  Improving  Disposition  Times 

From  this  research,  insight  into  factors  affecting  disposition  times  has  been  gained. 
Recommended  actions  to  shorten  disposition  times  are  provided  below: 

1.  The  DoD  cunently  tracks  disposition  times  from  the  latest  audit  report,  rather 
than  the  initial.  Because  the  average  time  between  the  initial  and  final  audit  was  calculated 
at  465  days.  DoD  disregards  well  over  a  year's  worth  of  work  spent  on  the  resolution  of 
each  defective  pricing  case.  This  lack  of  accountability  obscures  the  true  time  it  takes  to 
settle  a  defective  pricing  case  and  the  fact  that  settlement  times  are  considerably  worse 
than  reported.  Therefore,  the  researchers  recommend  tracking  disposition  times  from  the 
initial  audit  date  to  ensure  more  management  attention  is  placed  on  settling  defective 
pricing  cases  in  a  timely  manner. 

2.  To  reduce  the  number  of  supplemental  audits  and  RPA  revisions,  contractors  and 
contracting  officers  should  submit  their  comments  to  the  auditor's  preliminary  Hndings 
prior  to  the  completion  of  the  initial  audit  report.  Contractors  should  also  try  and  submit 
any  offsets  during  this  period  as  well.  In  many  of  the  case  Hies,  contractors  deferred 
comments  and  offset  proposals  until  after  the  release  of  the  audit  report  and  contracting 
officers  simply  did  not  review  the  preliminary  findings. 

3.  A  reporting  mechanism  is  needed  to  identify  when  fraud  cases  are  closed  because 
of  insufficient  evidence.  In  these  cases,  the  defective  pricing  allegation  still  requires 
settlement  by  the  contracting  agency. 
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4.  For  those  organizations  that  resolve  defective  pricing  cases  on  a  decentralized 
basis,  a  PCO  team  may  be  temporarily  assembled  to  settle  cases  which  are  extremely 
overage  or  can  no  longer  be  assigned  to  a  program  office  because  of  program  cancellation. 

5.  To  increase  the  priority  placed  on  defective  pricing  cases,  defective  pricing  funds 
should  be  returned  to  the  program  office,  instead  of  the  General  Treasury.  In  essence,  the 
program  office  is  being  penalized  for  contractor  defective  pricing.  Program  office  or 
centralized  defective  pricing  personnel  spend  their  time  recouping  overpayments,  but  those 
payments  are  not  returned  to  the  program  office. 

Suggestions  for  Further  Research 

1.  Because  this  research  was  limited  to  AFMC  product  centers,  the  findings  can  be 
generalized  only  for  those  centers.  The  researchers  recommend  a  smdy  of  AFMC's 
logistic  centers  to  determine  if  the  product  center  findings  are  valid  for  logistic  centers. 
Similarly,  the  same  research  could  be  conducted  on  operational  contracting  cases. 

2.  This  research  did  not  investigate  several  faaors  proposed  by  defective  pricing 
experts,  including  expertise  of  defective  pricing  participants,  workload,  and  priority. 
Investigation  was  not  accomplished  because  of  the  difficulty  in  collecting  data  or  the 
inability  to  determine  which  type  of  data  should  be  collected  to  accurately  measure  the 
factors.  Because  the  researchers  believe  these  factors  do  affect  sustention  rates  and 
disposition  tinKS,  further  analysis  of  the  omitted  factors  is  recommended. 
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Appendix  A:  Defective  Pricing  Survey 


Factors  Affecting  Defective  Pricing  Cases 

The  purpose  of  this  interview  is  to  identify  factors  affecting  sustention  rates  and 
disposition  times  in  defective  pricing  cases.  On  the  basis  of  several  General  Accounting 
Office  (GAO)  and  DoD  Inspector  General  reports  and  discussions  with  various  defective 
pricing  experts,  a  list  of  potential  factors  was  created.  From  this  list,  we  ask  you  to 
evaluate  the  effect  of  these  factors  on  sustention  rates  and  disposition  times. 

Definitions: 

Sustention  Rate  is  the  percentage  of  the  Defense  Contract  Audit  Agency’s  (DCAA’s) 
recommended  price  adjustment  (RPA)  in  its  last  audit  supplement  that  is  sustained  during 
negotiations  or  litigation. 

Disposition  Time  is  the  total  number  of  days  from  the  date  of  the  DCAA's  last  audit 
supplenoent  to  the  distribution  date  of  the  contract  modification. 

Scale: 

For  all  questions,  please  use  the  following  scale: 

1  2  3  4  5 

I _ I _ I _ I _ I 

No  Slight  Moderate  Strong  Very  Strong 

Effect  Effect  Effect  Effect  Effect 


U 

Unknown 

Effect 


Questions: 


CATEGORY  1:  Category  1  questions  are  associated  with  the  defective  pricing  action. 


1.1  What  effect  does  the  amount  of  alleged  defect  (the  auditor's  RPA)  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

1.2  What  effect  does  the  proportion  of  the  amount  of  the  alleged  defect  to  the  total 
audited  contract  action  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 
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1 .3  Defective  pricing  occurs  within  the  major  cost  elements  of  labor,  material,  overhead, 
general  and  administrative,  and  other.  What  effect  does  the  cost  element  in  which  the 
defect  occurred  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

1.4  Defective  pricing  occurs  at  the  prime  contractor  and  subcontractor  level.  What 
effect  does  the  contractor  level  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

1.5  Defective  pricing  occurs  on  two  types  of  contract  actions:  initial  contracts  and 
modifications  to  existing  contracts.  What  effect  does  the  type  of  contract  action  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

1.6  Contracting  officer  disagreement  with  the  RPA  primarily  occurs  when  the 
contracting  officer  determines  the  audit  report  to  be  inaccurate,  outdated,  inconclusive,  or 
unsupportable.  What  effect  does  the  contracting  officer's  disagreement  with  the  RPA  have 
on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

1.7  What  effect  does  the  number  of  defective  pricing  issues  addressed  in  the  audit  report 
have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

1.8  What  effect  do  proposed  contractor  offsets  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

1.9  Defective  pricing  cases  are  settled  through  (1)  negotiations,  (2)  contracting  officer 
final  decisions,  which  were  not  subsequently  litigated,  or  (3)  contracting  officer  final 
decisions,  which  were  subsequently  litigated  in  the  boards  or  courts.  What  effect  does  the 
method  of  disposition  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

1.10  What  effect  do  fraud  investigations  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  1  U 
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1.11  What  effect  does  the  length  of  time  to  settle  defective  pricing  cases  have  on 
sustention  rates? 

1  2  3  4  5  I  U 


CATEGORY  2:  Category  2  questions  are  associated  with  the  contractor  or  original 
contract. 


2.1  Defective  pricing  occurs  on  fixed-price  and  cost  reimbursement  contracts.  What 
effect  does  contract  type  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  1  U 

2.2  What  effect  does  the  total  amount  of  the  audited  contract  action  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

2.3  Contractor  size  is  defined  as  large  or  small.  What  effect  does  contractor  size  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

2.4  Contract  cost  performance  is  evaluated  as  an  underrun,  overrun,  or  on-target.  What 
effect  does  cost  performance  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

2.5  What  effect  does  the  identity  of  the  prime  contractor  (e.g.,  Boeing  vs.  Lockheed  vs. 
Hughes)  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 


CATEGORY  3:  Category  3  questions  are  associated  with  human  resources. 


3. 1  What  effect  does  government  negotiator  expertise  in  defective  pricing  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 

3.2  What  effect  does  contractor  negotiator  expertise  in  defective  pricing  have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 
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3.3  In  some  government  organizations,  defective  pricing  cases  are  dispositioned  by 
resources  dedicated  exclusively  to  defective  pricing.  In  others,  cases  are  absorbed  into  the 
cognizant  contracting  officer  workload.  What  effect  does  the  organizational  structure 
have  on 

a.  sustention  rates?  1  2  3  4  5  I  U 

b.  disposition  times?  1  2  3  4  5  I  U 


CATEGORY  4:  Category  4  questions  allow  for  additional  factors  to  be  suggested. 


4.1  What  factors  were  not  addressed  that  you  believe  affect  sustention  rates? 

4.2  What  effect  do  your  proposed  factors  have  on  sustention  rates?  1  2  3  4  5  I  U 

4.3  What  factors  were  not  addressed  that  you  believe  affect  disposition  times? 

4.4  What  effect  do  your  proposed  factors  have  on  disposition  times?  1  2  3  4  5  I  U 
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Name 


Cind 


Mike  Quinlin 


Bob  Williams 


Mai  Heniv  Gaudieau 


OrKanization/Position 


ASC  Focal  Point 


wmumm 


Eueoie  Solimine 


Ron  Schumann 


John  McPhearson 


Tom  Mohrhaus 


Scott  Gen 


Bill  Nedshen 


Pat  Herrick 


iinKBa 


Adonijah  Edwards 


Paul  Guntaiz 


Jim  McKenna 


DickC 


Dave  Hawkins 


Ed  Kerns 


Suzanne  Smds 


Michael  DiRenzo 


ASC  Contractine  Officer 


ASC  Focal  Point 


HO  AFMC  Focal  Point 


AFMCLC  Attome 


AFMCLC  Attome 


HO  IXTAA  Focal  Point 


Liaison  Auditor  @  ASC 


Liaison  Auditor  @  SMC 


Liaison  Auditor  @  ESC 


SMC  Focal  Point 


SMC  Negotiator 


SMC  Contracting  Officer 


ESC  Focal  Point 


ESC  Contracting  Officer 


ESC  Contracting  Officer 


ESC  Contracting  Officer 


ESC  Contracting  Officer 


DoDIn 


DoDIn 


DoD  In 


Interview  Date 


4/18 


4/15 


4/15 


4/15 


4/18 


4/15 


4/15 


4/22 


4/20 


4/25 


4/18 


4/20 


4/29 


4/19 


4/22 


4/20 


4/22 


4/22 


4/29 


4/29 


4/25 


4/20 


4/29 
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Appendix  D:  Additional  Factors  Suggested  During  Interviews 


The  additional  factors  suggested  during  the  interviews  can  be  categorized  and 
summarized  as  follows: 

Interest  Assessed.  The  1987  amendments  to  TINA  added  a  provision  for  interest 
assessment  to  encourage  prompt  resolution  by  the  contractor.  To  evaluate  the  impact  of 
this  provision,  a  categorical  variable  indicating  whether  or  not  interest  was  assessed  was 
included  f(»r  further  analysis. 

Audit  Characteristics.  Several  interviewees  stated  that  characteristics  of  the  audit 
itself  were  important  drivers  of  sustention  rates  and  disposition  times.  The  factors  that 
were  mentioned  include  the  quality  and  completeness  of  the  audit,  the  amount  of 
communication  and  cooperation  between  all  involved  parties,  and  the  willingness  of  the 
auditor  to  revise  his  initial  RPA.  While  direct  measurement  of  these  items  is  difficult,  the 
presence  of  RPA  Revisions  captures  the  essence  of  all  of  these  factors.  The  quality  and 
completeness  of  the  audit  is  also  captured  in  the  variable  reflecting  contracting  officer 
disagreement  with  the  auditor's  final  RPA.  As  such,  these  two  factors  combined,  revisions 
to  the  RPA  and  contracting  officer  disagreement  with  the  final  RPA,  encompass  the  above 
mentioned  suggestions. 

Subcontractor  Assist  Audits.  Another  audit  factor  suggested  was  the  number  of 
assist  audits  conducted  in  support  of  a  prime  contractor  audit.  Assist  audits  are  audits  on 
subcontractor  defective  pricing  that  are  incorporated  into  the  audit  on  the  prime  contract. 
The  number  of  assist  audits  is  difficult  to  ascertain  from  the  available  documentation  and 
this  factor  was  not  considered  for  analysis  of  variance  and  model  building.  In  addition,  the 
number  of  audit  assists  is  likely  to  be  highly  crarclated  with  the  craitractor  level  at  which 
the  defective  pricing  occurred  and  the  number  of  issues  involved,  both  of  which  are  factors 
being  analyzed. 
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Contractor  Attitudes.  Several  factors  related  to  the  contractor  were  proposed 
during  the  interviews.  Included  in  this  group  were  contractor  procrastination  in  settling 
cases,  willingness  to  settle,  agreement  with  the  RPA,  and  contractor  networking. 
Contractor  networking  is  the  communication  among  contractors  concerning  successful 
methods  of  resisting  defective  pricing  allegations.  All  of  these  factors  can  affect  the  speed 
with  which  the  contractor  is  willing  to  settle  the  case. 

Quality  Documentation.  The  quality  and  availability  of  government  evidence  to 
support  its  allegation  of  defective  pricing  can  have  a  significant  effect  on  disposition  times 
and  sustention  rates.  Lost  or  poorly  documented  files  severely  hinder  the  government's 
ability  to  defend  its  case.  This  factor  cannot  be  measured  with  available  data,  and  was 
therefore  not  considered  for  analysis  of  variance  and  model  building.  Nonetheless,  the 
importance  of  thorough  documentation  of  all  contract  actions  cannot  be  overemphasized. 

Expertise  of  Participants.  The  interview  results  showed  that  government  and 
contractor  negotiator  expertise  in  defective  pricing  were  significant  factors  for  both 
sustention  rates  and  disposition  times.  In  addition,  the  respondents  emphasized  the  effect 
of  all  participants  in  the  process.  Not  only  is  the  expertise  of  the  negotiators  important, 
but  so  are  the  expertise,  experience,  and  availability  of  all  support  personnel  (e.g.  buying 
team,  legal,  pricing,  defective  pricing  focal  points,  auditors,  technical  personnel,  and  audit 
liaisons).  The  resolution  of  a  defective  pricing  case  is  a  team  effort  and  the  outcome  is 
dependent  on  all  parties  involved.  As  with  negotiator  and  contractor  expertise,  direct 
measurement  of  these  factors  was  not  possible  with  the  available  data. 

Personnel  Turnover.  Personnel  turnover  can  affect  sustention  rates  and  disposition 
times  in  two  ways.  First  is  the  availability  of  personnel  involved  on  the  contract  action 
later  found  to  be  defectively  priced.  If  available,  these  individuals  can  provide  personal 
knowledge  of  what  transpired  during  negotiations.  This  familiarity  is  expected  to  increase 
sustention  rates  and  decrease  disposition  times.  The  second  factor  involves  personnel 
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turnover  during  the  time  a  defective  pricing  case  is  open.  The  need  for  new  personnel  to 
become  familiar  with  the  issues  is  expected  to  affect  both  rates  and  times.  Despite  its 
perceived  impact,  personnel  turnover  cannot  be  measured  with  available  data,  and  was 
therefore  not  considered  for  analysis  of  variance  and  model  building. 

Management  Attention.  The  management  attention  placed  on  resolving  defective 
pricing  cases  was  suggested  as  a  significant  factor  by  several  interviewees.  Management 
attention  can  be  demonstrated  in  a  number  of  ways,  including  training,  assignment  of  focal 
points,  and  regular  reporting  of  case  information  to  upper  management.  It  can  also  be 
demonstrated  at  a  variety  of  levels  such  as  the  local  Director  of  Contracting,  Major 
Command,  Headquarters,  Air  Staff,  GAO,  and  IG.  Management  also  sets  the  attitude 
toward  defective  pricing  and  can  influence  whether  defective  pricing  is  perceived  as  an 
administrative  irritant  or  an  important  problem.  As  with  the  related  factor  involving  the 
priority  given  to  defective  pricing,  the  breadth  and  depth  of  management  attention  make 
objective  measurement  virtually  impossible.  Investigation  into  the  differences  between 
centralized  and  decentralized  offices,  however,  may  provide  some  insight 
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Appendix  E:  Variable  Descriptions 


The  following  is  a  description  of  the  data  collected  to  represent  the  factors  identified 
in  Table  2  for  further  evaluation.  The  variable  nait^  used  in  the  computer  analyses  is 
included  in  parentheses. 

Alleged  Defect  Amount  (RPA).  This  research  was  interested  in  the  effect  of  the 
magnitude  of  the  RPA  on  both  sustention  rates  and  disposition  times.  A  difficulty  was 
encountered,  however,  because  sustention  rate  is  calculated  by  dividing  the  total  dollars 
recovered  by  the  RPA.  Since  the  RPA  is  part  of  the  equation  for  sustention  rate,  there  is, 
by  definition,  a  high  correlation  between  the  two.  To  overcome  this,  the  RPA  was 
measured  on  a  categorical  basis  as  follows:  (1)  RPA  less  than  $2(X),0(X)  (2)  RPA 
greater  than  or  equal  to  $200,000  but  less  than  $1,000,000  and  (3)  RPA  greater  than  or 
equal  to  $1,000,000. 

Cost  Element  (COSTELEM).  The  cost  elements  involved  in  the  defective  pricing 
cases  were  categorized  as  follows:  Labor,  Material,  Other,  Multiple.  Categories  such  as 
General  and  Administrative  and  Oveihead  were  combined  into  the  Other  category  because 
of  the  small  number  of  data  points. 

Contractor  Level  (PRIMESUB).  This  factor  identifies  which  party,  the  prime 
contractor  or  subcontractor,  committed  the  defective  pricing  If  the  prime  contractor 
failed  to  submit  its  own  data  or  properly  disclose  subcontractor  data,  the  defect  was  held 
to  be  at  the  prime  contractor  level.  If  the  case  involved  data  not  properly  submitted  to  the 
prinic  by  the  subcontractor,  the  defect  was  held  to  be  at  the  subcontractor  level.  The  one 
case  which  involved  defective  pricing  by  both  the  subcontractor  and  the  prime  contractor 
was  categcnized  as  subcontractor. 

CO  Disagreement  with  the  RPA  (DISAGREE).  Contracting  Officer 
disagreement  with  the  audit  report  was  assessed  through  a  comparison  of  the  CO's 
negotiation  objective  with  the  final  RPA.  An  objective  of  less  than  the  RPA  indicates 
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disagreen^nt.  The  categorical  variable  DISAGREE  indicates  whether  or  not  this  situation 
hasoccuired. 

Number  of  Issues  (ISSUES).  This  variable  recorded  whether  the  defective  pricing 
case  involved  single  or  multiple  issues.  Categorical  rather  than  real  number  data  were 
used  because  of  the  insufficient  number  of  data  points  within  each  real  number  category 
(e.g.,  2  issues,  3  issues). 

Legal  Complexity  (GRAY).  GRAY  was  an  indicator  variable  which  specified 
whether  or  not  the  legal  issues  involved  were  complex  or  not.  Determining  factors 
included  the  presence  of  opinions  commenting  on  the  legal  sufficiency  of  contractor  and 
audimr  rebuttals,  documented  evaluation  of  litigation  risk,  strong  contractor  defenses  to 
(xie  of  the  required  proof  elements,  and  an  overall  assessment  of  the  legal  difficulties  in 
meeting  the  burden  of  proof  in  each  case. 

Offsets  (OFFSETS).  This  categorical  variable  identified  whether  offsets  were 
proposed  and,  if  so,  how  they  were  evaluated.  The  categories  were:  (1)  No  offsets 
proposed,  (2)  Offset(s)  proposed  and  fully  accepted  by  the  Contracting  Officer,  (3) 
Offset(s)  proposed  and  partially  accepted  by  the  Contracting  Officer,  and  (4)  Offsets) 
proposed  and  not  accepted  by  the  Contracting  Officer. 

Method  of  Disposition  (METHDISP).  This  variable  categorizes  cases  according 
to  whether  or  not  they  were  litigated.  Further  breakdown  into  categories  such  as 
negotiated,  final  decision  not  appealed,  final  decision  appealed  but  case  settled  out  of 
court,  etc.  was  infeasible  due  to  the  small  number  of  data  points  in  each  of  the  smaller 
categories. 

Fraud  Investigation  (FRAUD).  FRAUD  was  an  indicator  variable  which  recorded 
whether  or  not  a  given  case  was  investigated  fex  fraud. 

Disposition  Time  (TIME).  As  an  independent  variable,  disposition  time  was 
measured  on  a  categorical  basis  as  follows:  (1)  Disposition  time  less  than  or  equal  to  1 
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year,  (2)  Disposition  time  greater  than  1  year  and  less  than  or  equal  to  2  years,  (3) 
Disposition  time  greater  than  2  years  and  less  than  or  equal  to  3  years,  and  (4)  Disposition 
time  greater  than  3  years. 

Identity  of  the  Prime  Contractor  (KTR).  Contractors  involved  in  ten  or  more 
cases  were  investigated  separately.  Separate  categories  were  established  for  each  of  the 
following  contractors:  Boeing,  General  Dynamics,  Loral,  Martin  Marietta,  McDonnell 
Douglas,  and  United  Technologies/Pratt  &  Whitney.  All  others  were  combined  into  one 
category.  Such  grouping  was  necessary  because  it  is  not  statistically  meaningful  to 
evaluate  a  contractor  with  only  a  few  data  points.  A  fair  amount  of  history  with  a 
particular  contractor  is  necessary  to  accurately  assess  that  contractor's  impact  on  the 
dependent  variables. 

Product  Center  (CENT).  The  interviews  identified  organizational  structure  as  an 
important  factor  for  both  sustention  rates  and  dispositicm  times.  ASC  handles  defective 
pricing  cases  on  a  centralized  basis  while  ESC  and  SMC  handle  cases  on  a  decentralized 
basis.  The  effect  of  the  organizational  structure  will  be  assessed  through  the  use  of  this 
variable  which  indicates  which  center  handled  the  case.  If  there  is  a  difference  between 
centralized  and  decentralized  offices,  there  should  be  significant  differences  between  both 
ASC  and  ESC,  and  ASC  and  SMC.  Evaluating  based  on  the  specific  product  center 
involved  helps  to  isolate  the  effect  of  the  organizational  structure  from  the  other  potential 
differences  between  the  centers. 

Interest  (INTEREST).  This  variable  indicated  whether  or  not  interest  was 
assessed. 

RPA  Revisions  (RPA_REVS).  This  was  an  indicator  variable  which  recorded 
whether  or  not  the  auditor  revised  the  RPA. 
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Appendix  F:  Summary  of  Subset  Selection  Results  Using  Miller's  Method 


SUSTENTION  RATES 


INITIAL  SCREENING  (includes  all  variables  identified  through  ANOVA ): 


Selection 

Rate  Variables  Selected  In  Oder  of  Appearance 

50%  DISAGREE/GRAY/METHDISP/RPA_LRG/CENT_ESC/KTR_MA 

40%  DISAGREE/GRAY/METHDISP/RPA_LRG/CENT_ESC/KTR_MA/OFF_PAR 

10%  DISAGREE/GRAY/METHDISP/RPA_LR 

FINAL  SCREENING  (includes  the  7  variables  selected  ftom  the  initial  screening) 

60%*  DISAGREE/GRAY/METHDISP/RPA_LRG/CENT_ESC/KTR_MA 

40%  DISAGREE/GRAY/METHDISP/RPA_LRG/CENT_ESC/KTR_MA/OFF_PAR 

I  DISPOSITION  TIMES 


INITIAL  SCREENING  (includes  all  variables  indentified  through  ANOVA ): 


Selection 

Ratg  Variables  Selected  In  Order  of  Appearance 

40%  GRAY/RPA_REVS/CENT_ESC/KTR_MC/RPA_LRG 

30%  GRAY/RPA_REVS/CENT_ESC 

20%  GRAY/RPA_REVS 

10%  GRAY/RPA_REVS/CENT_ESC/KTR_MC/RPA_LRG/KTR_BO/METHDISP/ 

KTR_MA 


FINAL  SCREENING  (includes  the  5  variables  selected  from  the  initial  screening) 

90%*  GRAY/RPA_REVS/CENT_ESC/KTR_MC/RPA_LRG 

10%  GRAY/RPA_REVS/CENT_ESC 

*  Model  Selected 
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